PowerShell V4 – PipelineVariable Common Parameter

The big new feature in Windows PowerShell 4.0 is Desired State Configuration however there are a number of other minor features one of which is a new common parameter called PipelineVariable.  This parameter allows you to store the current pipeline object into the variable specified by the name you provide.  This capability comes in handy when, in the process of executing commands in the pipeline, information is lost due to transformations on the objects flowing down the pipeline.  For example, take the following case:

PS> Get-ChildItem *.ps1 | Select-String function | Get-Member

   TypeName: Microsoft.PowerShell.Commands.MatchInfo

Name         MemberType Definition
----         ---------- ----------
Equals       Method     bool Equals(System.Object obj)
GetHashCode  Method     int GetHashCode()
GetType      Method     type GetType()
RelativePath Method     string RelativePath(string directory)
ToString     Method     string ToString(), string ToString(string directory)
Context      Property   Microsoft.PowerShell.Commands.MatchInfoContext Context {get;set;}
Filename     Property   string Filename {get;}
IgnoreCase   Property   bool IgnoreCase {get;set;}
Line         Property   string Line {get;set;}
LineNumber   Property   int LineNumber {get;set;}
Matches      Property   System.Text.RegularExpressions.Match[] Matches {get;set;}
Path         Property   string Path {get;set;}
Pattern      Property   string Pattern {get;set;}

Note in this output that the FileInfo objects output by Get-ChildItem got replaced with MatchInfo objects in the pipeline.  Most of time this is perfectly fine except when you’d like to access some information that was available on a pipeline object in a preceding stage of the pipeline. 

An example of this kind of scenario comes from a question on StackOverflow, the OP wanted to know how to print out the filename and just the immediate parent directory’s name e.g. output “Scripts\foo.ps1” instead of “C:\Users\Keith\Scripts\foo.ps1”.  This can be accomplished with Split-Path and Join-Path manipulation of the MatchInfo’s Path property e.g.:

Get-ChildItem *.ps1 | Select-String function | 
    Foreach {"$(Split-Path (Split-Path $_ -Parent) -Leaf)\$($_.Filename)"}


This works just fine but being a clever PowerShell user you happen to know that the FileInfo objects output by Get-ChildItem have a convenient property called “Directory.BaseName” that would be perfect in this scenario.  Unfortunately you quickly realize that after Select-String you no longer have access to that FileInfo object.  Select-String outputs a new type of object called Microsoft.PowerShell.Commands.MatchInfo. You could make the FileInfo object available via another variable ($fi) using the Foreach command as shown below:

Get-ChildItem *.ps1 | Foreach {$fi = $_; $_} |
    Select-String function | 
    Foreach {"$($fi.Directory.BaseName)\$($_.Filename)"}


The new PipelineVariable allows you to eliminate the extra Foreach command.  The example below effectively does the same thing as the previous example:

Get-ChildItem *.ps1 -PipelineVariable fi | Select-String function | 
    Foreach {"$($fi.Directory.BaseName)\$($_.Filename)"}


There are a few things to note about the PipelineVariable parameter.  First, it has a conveniently short alias “pv” just like the other common parameters tend to have e.g. “ea” for ErrorAction, “ov” for OutputVariable, “wv” for WarningVariable.  Second, you do not specify a “variable” as the value for a PipelineVariable parameter. Instead you specify the name of a variable.  In other words, do not do this: “-PipelineVariable $fi” unless you really want to programmatically provide the name of another variable in which to store the pipeline object.  Most of the time you will not specify the “$” but just the name of a variable e.g. “-PipelineVariable fi”.  Third, using the PipelineVariable parameter works great as long as all the commands in the pipeline between the storage and the use of the PipelineVariable are streaming commands.  If you use Sort-Object or Group-Object in between, then the object your pipeline variable refers to will not be correct because the way these commands work.  Commands like Sort-Object and Group-Object aggregate all pipeline input, process it and then start a new flow of objects down the pipeline. 

Here’s a simple example that demonstrates the problem. We have the following filenames and sizes:

PS> ls *.txt
Mode           LastWriteTime       Length Name
----           -------------       ------ ----
-a---     7/19/2013 12:16 AM     20000006 Large.txt
-a---     7/19/2013 12:16 AM       200006 Medium.txt
-a---     7/19/2013 12:16 AM         2006 Small.txt
-a---     7/19/2013 12:16 AM           26 VerySmall.txt


Let’s try to sort the files based on length while using the PipelineVariable parameter.  Yes I know, completely contrived but it demonstrates the point.

PS> Get-ChildItem *.txt -pv fi | Sort-Object Length |
>>     Foreach {"$($_.Name) size is $($fi.Length)"}
VerySmall.txt size is 26
Small.txt size is 26
Medium.txt size is 26
Large.txt size is 26


This is obviously not correct.  What happens is that the Foreach scriptblock doesn’t execute until Get-ChildItem has output *ALL* its objects.  That’s because the Sort-Object command buffers up all its input before it sends any output down the pipeline.  So by the time the Foreach begins to execute, the pipeline variable $fi will always contain the last object output by Get-ChildItem.  If you think about it this makes sense.  Sort-Object can’t send any “sorted” output down the pipeline until it sees all its input.  The issue happens with any non-streaming command which also includes Group-Object and Measure-Object.

Finally, one last example where this feature comes in handy.  This has to do with using Select –ExpandProperty which is a great way to walk down collections of collections.  The issue I’ve run into doing this is sometimes I want to preserve a property from the collection’s parent object but can’t because there is an identically named property on the items in the collection.  Here’s an example:

PS> Get-Command -Type Cmdlet | Select Name -ExpandProperty ParameterSets
Select : Property cannot be processed because property "Name" already exists.


In other words, I can’t include the Name from the CmdletInfo object because it would conflict with the ParameterSet.Name property. This means you have to use the Foreach trick I showed earlier:

Get-Command -Type Cmdlet | Foreach {$cmd=$_;$_} | Select -ExpandProperty ParameterSets


But if you then go on to expand the Parameters collection in each ParameterSet you run into the same problem again since each Parameter object also has a Name property.  Rather than injecting another Foreach command into the pipeline, this can be simplified using the PipelineVariable parameter in PowerShell 4.0 like so:

Get-Command -Type Cmdlet -pv cmd | 
    Select -Exp ParameterSets -pv pset |
    Select -Exp Parameters |
    Format-Table @{n='Cmdlet';e={$cmd.Name}}, @{n='ParameterSet';e={$pset.Name}}, Name


BTW you may be tempted to use a feature of Select-Object where you can add synthetic properties to objects in the collection as PowerShell expands the collection like so:

Get-Command -Type Cmdlet | Select @{n='Cmdlet';e={$_.Name}} -ExpandProperty ParameterSets


This works great the first time you execute it.  But upon subsequent executions, you get an error indicating that the property Cmdlet already exists.  I believe this happens because in PowerShell 3.0 this approach modifies the actual objects via the Dynamic Language Runtime. This is fine except when those objects are cached i.e. not regenerated every time they are asked for.  When that happens, you get an error because you are trying to add the same property again to the same objects you previously added the property to. 

That wraps up this discussion on the new PipelineVariable parameter in PowerShell 4.0.  I expect this parameter will save you some typing in scenarios like the ones described above.  Just watch out for aggregating commands like Group-Object and Sort-Object.

This entry was posted in PowerShell, PowerShell 4.0. Bookmark the permalink.

13 Responses to PowerShell V4 – PipelineVariable Common Parameter

  1. Pingback: Microsoft Most Valuable Professional (MVP) – Best Posts of the Week around Windows Server, Exchange, SystemCenter and more – #38 - TechCenter - Blog - TechCenter – Dell Community

  2. Pingback: Server King » Dell’s Digest for July 22, 2013

  3. Pingback: What’s new in the SQL Server PowerShell in the SQL Server 2014 CTP1 ? | $hell Your Experience !!!

  4. Pingback: O que tem de novo no SQL Server PowerShell no SQL Server 2014 CTP1 | $hell Your Experience !!!

  5. Steve says:

    Can you describe the difference between the new PipelineVariable and the older OutputVariable?
    I can substitute OutputVariable in your example and get the same result (for the limited tests I tried). OutputVariable also works for debugging various steps in a pipeline if you use different variables for each step.

    • rkeithhill says:

      The difference is that -OutputVariable accumulates that output at the associated command outputs more than one object. So if your limited test only output a single object you would see the same results. But if the command outputs multiple objects then the variable used with -OutputVariable will contain a collection of objects. The variable used with -PipelineVariable will only ever contain the current pipeline object.

  6. Simon says:

    There’s another little trick it can be handy to have up your sleeve that can help with this kind of thing: when you put an assignment in brackets, the result of the assignment is output (and so gets passed down the pipeline). For example, everywhere you’ve used an expression like this:

    ... | Foreach {$cmd=$_;$_} | ...

    you could have written:

    ... | Foreach {($cmd=$_)} | ...

    Not much of a saving in this case, I admit, but it can often mean you don’t need an extra step in the pipeline. For example, your example:

    Get-Command -Type Cmdlet | Foreach {$cmd=$_;$_} | Select -ExpandProperty ParameterSets

    could have been written as:

    Get-Command -Type Cmdlet | Foreach {($cmd=$_).ParameterSets}
  7. Pingback: PowerShell 4.0 PipelineVariable, With or Without, I can´t live…..Without you. | $hell Your Experience !!!

  8. Pingback: Get-EsxCli on steroids - The Crazy Consultant

  9. Pingback: Powershell with a dollop of Chocolatey Sauce = Delicous

  10. Pingback: Automating Index Fragmentation Reports - Hey, Scripting Guy! Blog - Site Home - TechNet Blogs

  11. Pingback: A few PowerShell tips – Part 2 – Useful commands / switches. | Polisz Admin Blog

  12. Pingback: SAPIEN Technologies » Blog Archive » Friday Puzzle: What does a Catch block catch?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s