Make file redirections apply only to external commands

[IanManske]

Description

File redirections like out> and err> refer to the stdout and stderr streams of external commands/processes. However, the out>, out+err>, out>>, and out+err>> file redirections also work with values and list streams, saving any values to the file. This is despite the fact that nushell values and list streams do not quite have a defined "stdout" or "stderr". Currently, the pipe redirections (e>| and o+e>|) only work on external commands, erroring if used on a value or list stream.

ls o+e>| get name | save test.txt # errors

User-Facing Changes

Breaking change: o>, o>>, o+e>, and o+e>> now only apply to external commands.

[fdncred]

Why can't we make it work with internal commands too?

[IanManske]

@fdncred Are you referring to the file redirections (o>, etc.), the pipe redirections (e>|, etc.), or both?

[fdncred]

Yup. Seems like a UX problem if people have to remember that redirection only works for externals. Seems like we'd want to have the same experience whether it's internal or external.

But not necessarily for this PR. I'm fine with incremental moves.

[WindSoilder]

I'm not sure about it either. Because sometimes it's easy to do something like this: "aaa" o> file.txt, which is shorter than "aaa" | save -fw file.txt

[IanManske]

IMO, all redirections should either work with internal streams and values, or they all should not work. Having some work and others not work is inconsistent. Following from this, the semantics with some redirections don't quite make sense with internal streams and values. E.g., I don't think 'text' e>| str upcase should be allowed.

Additionally, allowing file redirections on streams and values may create some "gotcha" situations.

def cmd [] {
    ^extern1
    ^extern2
    ^extern3 | str contains 'text' # returns a bool
}

# I want to log/redirect the stdout and stderr of the external commands to file,
# but still want to be able to access the return value
if (cmd o+e> log.txt) {
    print 'text found'
}

Edit: the file redirections also only currently work with string and binary values anyways and so are a little limited.

[fdncred]

I agree that we need to be consistent in everything like this, but I would push that more towards being able to redirect stderr/stdout with internal and external commands as the way we should be consistent. Again, we can start making things consistent with this PR, but we need to have redirection work as well. I mean the entire reason we introduced redirection was to allow people's muscle memory to apply in nushell too. If we limit it to only externals, it kind of defeats the purpose IMO.

[fdncred]

Looking at someone's dotfiles I just noticed this, which I assume will stop working with this PR since http get isn't external. Is that correct or is it only for e>| and o+e>|?

 http get -r https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init o> rustup-init

[39555]

This pr introduces another strange difference between external and internal that I don't understand as a user. My more general question is: Why is there a user side difference if external command essentially is just a <string : "raw input"> type in nu pipeline (and raw input acts like a string implicitly in most cases)?

[39555]

More strange cases:

• help print says that print outputs a message to stdout or stderr, so what would happen with print "hello world" o>?
• initially I thought that error make {} would be in stderr but it's not true. I think stderr may also be structured and external stderr is just a 'raw input' type

[39555]

In that case i think may it work the same way as internal commands? discard all values and return the last one

def cmd [] {
    ^extern1
    ^extern2
    ^extern3 | str contains 'text' # returns a bool
}

[IanManske]

which I assume will stop working with this PR since http get isn't external.

That is correct. http get outputs a stream of values or a byte stream, and it does not interact with stdout. Previously, the redirection could also fail if the http get returns structured data (e.g., json).

Why is there a user side difference if external command essentially is just a <string : "raw input"> type in nu pipeline (and raw input acts like a string implicitly in most cases)?

External commands output a stream of arbitrary bytes, but string values in nushell are encoded as utf-8. So, the two are not necessarily compatible. Some commands assume utf-8 and try to split the external command output on lines, creating an error if this is not possible. Other commands collect all of the external command input into a string value (this may consume lots of memory if the external command output is large), and this will also create an error if utf-8 decoding fails. IMO, some of our commands should not be accepting external command output. E.g., what is each value in a stream of bytes?

help print says that print outputs a message to stdout or stderr, so what would happen with print "hello world" o>?

This is the one internal command where it might make sense to respect the current stdout and stderr destination. Currently, print always prints to nushell's stdout and stderr.

initially I thought that error make {} would be in stderr but it's not true. I think stderr may also be structured and external stderr is just a 'raw input' type

That is correct, nushell errors are structured values in the sense that they can be created (e.g., via error make). They can also be caught and inspected via try and catch. This part of what I meant by nushell values and list streams do not quite have a defined "stderr". Control flow can be interrupted by errors, but there is no concept of an error value channel.

And yes, external command stderr is a stream of bytes (unstructured data).

[sholderbach]

To me it seems that string and binary seem to move closer to the POSIX streams in their behavior with the things @IanManske and @devyn have been cooking. Which seems to make the UX better and also make the distinction harder.

On a related note there are some user assumptions around those redirections behaving more like POSIX e.g. in #12686

[IanManske]

Hmm, actually I think I might hold off on this for now.

1. because of the bytestream changes like @sholderbach mentioned.
2. I think redirections might need other changes, and I'd rather have a clearer picture before committing. I.e., this is a breaking change that might have other knock on effects for future designs and semantics.