Error: Windows stdio in console mode does not support writing non-UTF-8 byte sequences

[iyerusad]

Issue:

It seems 0.10 coreutils head/tail doesn't gracefully handle outputting UTF-16 content in Windows console. This is somewhat annoying since in PowerShell (versions prior to PowerShell 6.0), the redirection operators (>/>>) default to writing files in UTF-16LE.

Error: head.exe: error writing 'standard output': Windows stdio in console mode does not support writing non-UTF-8 byte sequences

PS C:\Users\User> echo "foobar" > test
PS C:\Users\User> cat test
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: test   <UTF-16LE>
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ foobar
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────
PS C:\Users\User> head test
C:\Users\User/.dotfiles/opt/bin\head.exe: error writing 'standard output': Windows stdio in console mode does not support writing non-UTF-8 byte sequences

PS C:\Users\User> tail test

thread 'main' panicked at src\uu\tail\src\tail.rs:431:33:
called `Result::unwrap()` on an `Err` value: Error { kind: InvalidData, message: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

PS C:\Users\User> echo $Host
Name             : ConsoleHost
Version          : 5.1.26100.4768
InstanceId       : 1c7962ab-05f7-417d-a559-f5dd21650b9a
UI               : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture   : en-US
CurrentUICulture : en-US
PrivateData      : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled  : True
IsRunspacePushed : False
Runspace         : System.Management.Automation.Runspaces.LocalRunspace

--==--
Testing with https://github.com/JavaScriptDude/cygtail (Cygwin) looks like it has gracefully output.

PS C:\Users\User\Desktop\cygtail-20220415> echo "foobar" > test
PS C:\Users\User\Desktop\cygtail-20220415> ./tail.exe test
□□foobar

Anticipated Result:

head/tail gracefully process files created by simple redirection output (e.g. echo "foobar" > test). Not sure if that means inserted unknown character glyphs or similar or converting on the fly to work within the presumed UTF-8 limit of windows console.

[ChrisDenton]

This is a consequence of Rust's standard library needing to convert to UTF-16 for console output and without knowing the encoding, UTF-8 is always assumed. The restriction on the encoding could be relaxed (e.g. by being lossy) but that requires changes in the standard library unless uutils wants to workaround it by manually doing console output.