Mixed Indentation breaks Windows shell commands

[xdhmoore]

Describe the bug
I've noticed that if I have nested indentation in a shell cmd string, even with the YAML >, the cmd doesn't get its newlines stripped as I would expect. When the string has newlines left within it, when it is run with sub_process, it seems to execute only the first line of the string. An example:

rules:
  - name: "test indentation"
    locations: .
    filters:
      - extension: 'yaml'
    actions:
      - shell: >
          pwsh -NoProfile -c "echo firstline;
          echo 'second line works'
          "
      - echo: "Shell out: {shell.output}"
      - confirm: "Above has 2 lines"

      - shell: >
          pwsh -NoProfile -c "echo firstline;
             echo 'indented second line doesnt work'
          "
      - echo: "Shell out: {shell.output}"
      - confirm: "Above is missing 2nd line"

I'm executing with organize run .\indent_config.yaml --working-dir .

Expected behavior
My understanding of the > operator is that it should remove all the newlines. And it does as long as you have uniform indentation for that string block. I can see in the logged output that in the second example the \n's are left in, while they are removed in the first example. I'm wondering if it might have something to do with this call to dedent.

The result is that scripted commands must be single-line, or fully-left indented.

Environment (please complete the following information):

• OS: Windows 10
• Output of organize --version: organize, version 2.2.0

[xdhmoore]

Actually, I think I'm misunderstanding the > operator. If you have extra indentation it seems to cause newlines to be preserved for that line, as seen on this page: https://yaml-multiline.info/ . That seems to be a YAML features. That said, I wonder if there's a way to pass it to subprocess that would execute more than the first line when there is mixed indentation.

[tfeldmann]

I'm not sure if I'm understanding this correctly. Couldn't you use | instead of >?

[xdhmoore]

I don't think so but I'll try it and get back to you. Sorry I didn't explain it very well. If I remember correctly the real problem is that sub_process only seems to execute the command string up to the first newline, so | preserving newlines wouldn't help. My issue above is that I thought I could remove all newlines with > but > doesn't always remove all the newlines depending on the indentation.

[xdhmoore]

I tried | with the above test and as I thought, it doesn't work because | retains newlines and what I need is a way to remove them. I said sub_process in my comments above, but sorry I meant subprocess.run(). It seems to only execute up to the first newline. Here's an example:

Example 1 - No newline:

python -c "import subprocess; from subprocess import PIPE; call = subprocess.run('''echo out 1 && echo out 2''', check=True, stdout=PIPE,stderr=subprocess.PIPE,shell=True,); print(call.stdout.decode('utf-8'))"

Output:

out 1
out 2

Example 2 - Same script except 1 newline in the command string:

python -c "import subprocess; from subprocess import PIPE; call = subprocess.run('''echo out 1
 && echo out 2''', check=True, stdout=PIPE,stderr=subprocess.PIPE,shell=True,); print(call.stdout.decode('utf-8'))"

Output:

out  1

I thought it might be an issue with reading the output, but I tried writing to a file before and after the newline, and it only worked before, so I think indeed the problem is that it only executes commands up to the newline.

[xdhmoore]

In the above examples I have an && between echo commands. Below I don't, so it's more like the use case for embedded scripts in yaml string blocks:

python -c "import subprocess; from subprocess import PIPE; call = subprocess.run('''echo out 1
 echo out 2''', check=True, stdout=PIPE,stderr=subprocess.PIPE,shell=True,); print(call.stdout.decode('utf-8'))"

Relevantly, his prints 1 line on windows but 2 on ubuntu, so I'm guessing it's some windows quirk.

[xdhmoore]

Since for me this is all about running powershell command strings, I found a workaround by setting the environment variable COMSPEC. The following changes the shell subprocess.run to use powershell:

$oldComSpec = $env:COMSPEC
try {
   $env:COMSPEC="$(Get-Command pwsh | select -ExpandProperty Source) -NoProfile"
   organize blah blah blah
} finally {
   $env:COMSPEC = $oldComSpec

}

This seems to fix the newline issue. Perhaps not a workaround for everyone, but so far this is working for me.