Effective PowerShell Item 10: Understanding PowerShell Parsing Modes

The way PowerShell parses commands can be surprising especially to those that are used to shells with more simplistic parsing like CMD.EXE.  Parsing in PowerShell is a bit different because PowerShell needs to work well as both an interactive command line shell and a scripting language.  This need is driven by use cases such as:

1. Allow execution of commands and programs with parameters at the command line.  Consequence: names (filenames, paths) should not require quotes unless there is a space in the name  (or path).
2. Allow scripts to contain expressions as found in most other programming/script languages.  Consequence: PowerShell script should be able to evaluate expressions like 2 + 2 and $date.Second as well as specify string using quotes e.g. “del -r * is being executed”.
3. Take code written interactively at the command line and paste it into a script for execution again at some point in the future. Consequence: These two worlds – interactive and script – need to coexist peacefully.

Part and parcel with providing a powerful scripting language is to support more types than just the string type.  In fact, PowerShell supports most .NET types including String, Int8, Int16, Int32, Decimal, Single, Double, Boolean, Array, ArrayList, StringBuilder among many, many other .NET types.  That’s very nice you say but what’s this got to do with parsing modes?  Think about this.  How would you expect a language to represent a string literal?  Well most folks would probably expect this representation: “Hello World”

And in fact, that is recognized by PowerShell as a string e.g.:

PS> "Hello World".GetType().Name
String
PS> "Hello World"
Hello World

And if you type a string at the prompt and hit the Enter key, PowerShell, being a very nice REPL environment, echoes the string back to the console as shown above.  However what if I had to specify filenames using quotes as shown below?

PS> del "foo.txt", "bar.txt", "baz.txt"

That would immediately “feel” different than any other command line shell out there.  Even worse, typing all those quotes would get really annoying, really fast.  What to do, what to do?  Well my guess is that the team, pretty early on, decided that they were going to need two different ways to parse.  First they would need to parse like a traditional shell where strings (filenames, dir names, process names, etc) do not need to be quoted.  Second they would need to be able to parse like a traditional language where strings are quoted and expressions feel like those you would find in a programming language.  In PowerShell, the former is called Command parsing mode and the latter is called Expression parsing mode. It is important to understand which mode you are in and more importantly, how you can manipulate the parsing mode.

Let’s look at an example.  Obviously we would prefer to type the following to delete files:

PS> del foo.txt, bar.txt, baz.txt

That’s better.  No bloody quotes required on the filenames.  PowerShell treats these filenames as strings even without the quotes in command parsing mode.  But what happens if my path has a space in it?  You would naturally try:

PS> del 'C:\Documents and Settings\Keith\_lesshst'

And that works as you would expect.  OK now what if I want to execute a program with a space in its path:

PS> 'C:\Program Files\Windows NT\Accessories\wordpad.exe'
C:\Program Files\Windows NT\Accessories\wordpad.exe

That didn’t work because are far as PowerShell is concerned we gave it a string, so it just echoes it back to the screen.  It did this because it parsed this line in expression mode.  We need to tell PowerShell to parse the line in command mode.  To do that we use the call operator ‘&‘ like so:

PS> & 'C:\Program Files\Windows NT\Accessories\wordpad.exe'

Tip: Help prevent repetitive stress injuries to your wrists and use tab (and shift+tab) completion for auto-completing the parts of a path.  If the resulting path contains a space PowerShell will insert the call operator for you as well as surround the path with quotes.

What’s going on with this example is that PowerShell looks at the first non-whitespace character of a line to determine which mode to start parsing in.  If it sees [_aA-zZ] or & or . or \ then PowerShell parses in Command mode.  One exception to these rules happens when the line starts with a name that corresponds to a PowerShell language keyword like “if”, “do”, “while”, etc.  In this case, PowerShell uses expression parsing mode and expects you to provide the rest of the syntax associated with that keyword.  The benefits of Command mode are:

• Strings do not need to be quoted unless there are embedded spaces in the string.
• Numbers are parsed as numbers and all other arguments are treated as strings except those that start with the characters: @, $, (, ‘ or “.  Numbers are interpreted as either Int32, Int64, Double or Decimal depending on how the number is decorated and the range required to hold the number e.g. 12, 30GB, 1E-3, 100.01d.

So why do we need expression parsing mode?  Well as I mentioned before it sure would be nice to be able to evaluate expressions like so:

PS> 64-2
62

It isn’t a stretch to see how some shells might interpret this example as trying to invoke a command named ’64-2′.  So how does PowerShell determine if the line should be parsed in expression mode?  If the line starts with a number [0-9] or one of these characters: @, $, (, ‘ or “ then the line is evaluated in expression mode. The benefits of expression mode are:

• It is possible to disambiguate commands from strings e.g. del -recurse * is a command whereas “del -recurse *” is just a string.
• Arithmetic and comparison expressions are straight forward to specify e.g. 64-2 (62) and $array.count -gt 100.  In command mode, -gt would be interpreted as a parameter if in fact the previous token corresponded to a valid command.

One consequence of the rules for expression parsing mode is that if you want to execute an EXE or script whose name starts with a number you have to quote the name and use the call operator e.g.:

PS> & '64E1'

If you were to attempt to execute “64E1” without using the call operator, PowerShell can’t tell if you want to interpret that as the number 64E1 (640) or execute the exe named 64E1.exe or the script named 64E1.ps1.  It is up to you to make sure you have placed PowerShell in the correct parsing mode to get the behavior you want which in this case means putting PowerShell into command parsing mode by using the call operator.  Note I have observed that if you specify the full command name e.g. 64E1.ps1 or 64E1.exe, it isn’t necessary to quote the command.

Now what if you want to mix and match parsing modes on the same line?  Easy.  Just use either a grouping expression (), a subexpression $() or an array subexpression @(). This will cause the parser to re-evaluate the parsing mode based on the first non-whitespace character inside the parens.

Sidebar: What’s the difference between grouping expressions (), subexpressions $() and array subexpressions @()?  A grouping expression can contain just a single statement.  A subexpression can contain multiple semicolon separated statements.  The output of each statement contributes to the output of the subexpression.  An array subexpression behaves just like a subexpression except that it guarantees that the output will be an array.  The two cases where this makes a difference are 1) there is no output at all so the result will be an empy array and 2) the result is a scalar value so the result will be a single element array containg the scalar value.  If the output is already an array then the use of an array subexpession will have no affect on the output i.e. array subexpressions do not wrap arrays inside of another array.

In the following example I have embedded a command “Get-ChildItem C:\Windows” into a line that started out parsing in expression mode.  When it encounters the grouping expression (get-childitem c:\windows), it begins parsing mode re-evaluation, finds the character ‘g’ and kicks into command mode parsing for the remainder of the text inside the grouping expression.  Note that “.Length” is parsed using expression mode because it is outside the grouping expression, so PowerShell reverts back to the previous parsing mode.  “.Length” instructs PowerShell to get the Length property of the object output by what was evaluated inside the grouping expression.  In this case, it is an array of FileInfo and DirectoryInfo objects.  The Length property tells us how many items are in that array.

PS> 10 + (get-childitem c:\windows).Length
115

We can do the opposite.  That is, put expressions in lines that started out parsing in command mode.  In the example below (admittedly lame) we use an expression to calculate the number of objects to select from the sequence of objects.

PS> Get-Process | Select -first (1.5 * 2)

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
-------  ------    -----      ----- -----   ------     -- -----------
    120       4    11860       9508    46            1320 audiodg
    778       6     1772       3516    88             560 csrss
    922      14     5288      13696   163             620 csrss

Using the ability to start new parsing modes, we can nest commands within commands.  This a powerful feature and one I recommend mastering.  In the example below PowerShell is happily parsing the command line in command mode when it encounters ‘@(‘ a.k.a. the start of an array subexpression.  This causes PowerShell to re-evaluate the parsing mode but in this case it finds a nested command.  One that grabs the new filename from the first line of the file to be renamed.  I used the array subexpression syntax in this case because it guarantees that we will get an array of lines even if there is just one line.  If you use a grouping expression instead and the file happens to contain only a single line then PowerShell will interpret the [0] to be “get me the first character in the string” which is “f” in the example below.

PS> Get-ChildItem [a-z].txt | Foreach { Rename-Item $_ -NewName @(Get-Content $_)[0] -WhatIf }
What if: Performing operation "Rename File" on Target "Item: C:\a.txt Destination: C:\file_a.txt".
What if: Performing operation "Rename File" on Target "Item: C:\b.txt Destination: C:\file_b.txt".

There is one final subtlety that I would like to point out and that is the difference between using the call operator (&) to invoke commands and “dotting” commands. Consider invoking a simple script that sets the variable $foo = ‘PowerShell Rocks!.  Let’s execute this script using the call operator and observe the impact on the global session:

PS> $foo
PS> & .\script.ps1
PS> $foo

Note that using the call operator invokes the command in a child scope that gets thrown away when the command (script, function, etc) exits.  That is, the script didn’t impact the value of $foo in the global scope.  Now let’s try this again by dotting the script:

PS> $foo
PS> . C:\Users\Keith\script.ps1
PS> $foo
PowerShell Rocks!

When dotting a script, the script executes in the current scope.  As a result, the variable $foo in script.ps1 effectively becomes a reference to the global $foo when the script is dotted from the command line resulting in changing the global $foo variable’s value.  This shouldn’t be too surprising since “dot sourcing”, as it’s also known, is common in other shells.  Note that these rules also apply to function invocation.  However for external EXEs it doesn’t matter whether you dot source or use the call operator since EXEs execute in a separate process and can’t impact the current scope.

Here’s a handy reference to help you remember the rules for how PowerShell determines the parsing mode.

First non-whitepace character	Parsing mode
[_aA-zZ], &, . or \	Command
[0-9], ‘, “, $, (, @ and any other character that doesn’t start command parsing mode	Expression

Once you learn the subtleties of these two parsing modes you will be able to quickly get past those initial surprises like how you execute EXEs at paths containing spaces to putting these parsing modes work for you.