Powershell tips for bash users, part 2, Day to day scripting

Photographer: dan http://www.freedigitalphotos.net/images/Workshop_and_DIY_g191-Woodplane_p7810.htmlOne of the best things in shells is that you can do some repetitive tasks quickly, efficiently and painlessly. For example, you have to replace some string with another string in multiple files scattered through different directories. You can do that using GUI editors, but this proces is not quick, it is dubiously efficient and most certainly is not painless. Since this is task most administrators and testers will often have, it is useful to have a boilerplate script or function which will handle it and which you can call from your other scripts or directly depending on situation.

In bash, I would do something like this


#!/bin/bash
find="$1"
replace="$2"
path="$3"
find $path | xargs grep -ls $find > matchingfiles.txt
LIST=`cat matchingfiles.txt |tr \n" " "`
for i in $LIST
do
sed "s|$find|$replace|g" $i > $i.new
mv $i.new $i
done

Script should be called with 3 parameters. 1st – the string to replace, 2nd – replacement string, 3rd – root directory to start search in.In order to make things short and easy readable, this example lacks some of good parts – usage guide, check existence of files… However, example works fine for the purpose.

Powershell

How to do similar thing in powershell? Here is the barebone equivalent:

$find=$args[0]
$replace=$args[1]
$path=$args[2]
(get-childitem $path -Recurse | where { ! $_.PSIsContainer }) | select-string -pattern $find | % {$_.Path} > matchingfiles.txt
$FILESARRAY = get-content matchingfiles.txt
foreach ($FILE in $FILESARRAY)
{
(get-content $FILE ) |foreach-object {$_ -replace $find, $replace} | set-content $FILE
}

Differencies? There are many. Apart from different syntax, there are fundamental things:

1. Finding things. “Get-Childitem” cmdlet is much more powerfull than “find”. You can use it also for traversing through registry and a certificate store, not only directories.2. Selecting only files. In bash we used “-l” switch for grep command. In powershell the easiest way I know is that we say that we do not want directories with “where { ! $_.PSIsContainer}”

3. After we formatted output (list of files), we now look for the string we want to replace. Select-String used here is similar enough to grep.

4. To be sure that pipeline will output only filenames we added a “foreach {$_.Path}” to the end identifying that we want only that member of the returned object. In bash, “grep –l” was enough.

5. In powershell script we added a content of the file in array, not in list as in bash. We could use an array in bash but this would broke the compatibility with sh.

6. We used pipeline inside foreach loop to modify files as oppose to straight editing with sed. Again a bit of overhead in processing.

7. With using brackets parenthesis arround (get-content $FILE) we assured rest of the pipeline will execute after this expression is completed, so we could write to the same file not using temp one we used with bash.

8. Big difference from bash is object orientation of powershell. More about that in following chapter.

Too much power for shell?

One thing for sure, a learning curve for powershell is a bit steeper than for bash. In bash you more or less dig through information in quick and dirty manner. Everything you get as output of your commands is pure text. You do not have to worry about objects and their members. Btw, try this in your PS command prompt:

PS> ls | Get-Member

You will see that powershel will display object members of every distinct object returned by “ls” command. This command we used as a tool to snatc those members we operated on in the above script. “.PSIsContainter” – which we evaluated to see is thephoto by Patrick Q returned object directory or “.Path” which was called get the path of returned file objects. Remember “Get-Member” command. You will find it useful. Shorthand for it is “gm.”

There is practically no way to write a script even as simple as this one without using objects. This can seem as a huge overhead for simple scripting tasks especially for someone who is not used to object oriented programming languages or someone who prefer quick and dirty or just simple approach.

So, does object orientation means powershell is more powerful than bash? It certainly does not mean that you can do more with less code. However, one of the goals powershell creators had in mind was to provide a way to access all system resources in similar manner. This should make life of admins, testers and other people who do automation task on Windows much easier. We will try to look at this claim in one of next articles in series.

  • Excelentes comentarios. Excelentes posts, nos vemos!

  • Honeymonster


    param( [string]$Find, [string]$Replace, [string]$Path )
    gci -Path $Path -recurse | foreach{ (gc $_) -replace $Find,$Replace | sc $_ }

    That should do the trick in PowerShell. If you save this to a file called myreplace.ps1 you will be able to call it using e.g.

    ./myreplace foo bar *.txt

    or

    ./myreplace -path *.txt foo bar

    Note how:

    PS allows you to define named parameters.
    PS allows you to optionally define typed parameters
    The advanced type system allows you to combine a pipeline without resorting to error-prone temporary files or strange brittle “list” variables. One line.

  • georgh

    Thanks for your post nd tips, I am a powershell newbie and it was very helpful.

    When doing this things in unix, I found the grep -r very useful, as it searches recursively and -l returns the mathcing file names. You can also pass -i option to sed to do the replace in-place. An empty parameter is passed to -i so no backups are created.


    grep -lr $find $path | xargs sed -i'' "s|$find|$replace|g"

    If you still want to keep track of the matched files you can use tee, which writes to a file and to standard output:


    grep -lr $find $path | tee matchingfiles.txt | xargs sed -i'' "s|$find|$replace|g"

  • asmoore82

    Your Bash script is terrible. It buries the entire article under a big pile of salt.

    #1 don’t reference variables without quoting them unless you have and understand a very good reason why not.

    #2 don’t pipe find to xargs without -print0 | -0

    #3 don’t reference variables without quoting them

    #4 don’t use backticks, they are fugly and do not nest, use $( … ) instead

    #5 don’t reference variables without quoting them

    #6 don’t use sed followed by mv, use sed -i instead

    #7 don’t reference variables without quoting them

    Good lord, your script has an unholy number of bugs that can LEAD TO DATA LOSS WITHOUT WARNING just by not quoting your references. Filenames can easily have spaces in them and your script tranples all over them without warning or confirmation.

    don’t do something like this: > $i.new

    What happens with $i is “very_important_file with_space_in_name” ??

    then it becomes: > very_important_file with_space_in_name.new

    and then very_important_file is toast.

  • asmoore82

    Let’s fix your Bash mess, better late than never!

    Any time you use for i in $… it’s a red flag you are doing something horribly wrong.

    grep supports recursive searches with -r;
    grep also supports -Z for null delimited output for piping results striaght to xargs -0;
    and sed supports multiple files names to update in-place with -i;

    #!/bin/bash

    # USAGE: replace.sh [find] [replace] [path]

    grep -rlsZ “$1” “$3” | xargs -0 echo sed -i “s|$1|$2|g”

    #End of File

    Note that this version only shows the command that would be run. Remove “echo” to make it really do it.

    Your extremely buggy script was just recuded to a bug-free one line script. Powershell Who?

  • asmoore82

    P.S. Also note it has the limitation of the pipe “|” not being usable in the search or replace strings.

  • asmoore82

    I previously refrained from commenting on any Powershell code. But now, at least for posterity, I think I have made enough sense out of Honeymonster’s code to point out the senselessness of it…

    What’s with the *.txt? That’s not the blueprint for how the replace is supposed to work; it is supposed to recurse a path, not operate on a globbed filename pattern. This is even more strange on your part when you yourself call it up like “-path *.txt” — *.txt is obviously not a path.

    Actually gci with *.txt as Path will return absolutely nothing. You must explicitly pass something like *.txt in as -Filter but not -Path.

    Which brings us to the “-path” usage overall. Is this supposed to be a good thing? An all lowercase arugument on the command line magically becomes a Capitalized Variable name in the script. Very, very sloppy indeed – just as I would expect from Powershell.

    And most importantly, what if the directory structure we are recursing has tens of thousands of files (10,000’s) or more but only 3 contain the text that needs replacing. Only those 3 should be targeted and changed/overwritten. How many files will be re-written to disk in your Powershell 1liner? All 10,000’s of them I do believe. Big difference.

    PS for extra credit… anyone who truly understands filename glob matching should also spot that the construct of “-path *.txt” is rubbish for a whole different reason. Proper shells do not pass glob patterns through to commands, unless explicitly escaped the pattern is immediately substituted for the real filenames. BIG difference there too. Every Unix command ever written doesn’t have to link in some library for filename globs; it’s already done for them at the shell level.

  • Pingback: xmt85c4wx5ctwxw3tcerthve56()

  • Pingback: http://falschgeldkaufen.blogspot.com/2017/01/wo-kann-ich-falschgeld-kaufen.html()