Translate ANSI characters

[Ambrevar]

Many shell commands leverage the terminal processing of special characters like ^H (backspace) and ^M (newline) to perform some special formatting, like rewriting text in place.

This is often seen when a progress bar is displayed by the command. Try rsync, wget, curl, your package manager...

Running these commands in a Lisp REPL often makes for pages of garbage output, making them practically un-runnable.

An easy way around this is to ask uiop:launch-program to write the output to a stream, then to process the stream for special characters.

Here is an example that processes ^H and ^M:

(let ((process (uiop:launch-program "test.sh" :output :stream :error-output :output)))
  (labels ((decode-process-output-to-stream (process output-stream)
             (alex:when-let ((line (read-line (uiop:process-info-output process) nil nil)))
               (let ((lines (str:split "
" line)))
                 (write-string
                  (sera:ensure-suffix
                   (str:join
                    (string #\n ewline)
                    (mapcar (lambda (line)
                              (reduce (lambda (&optional string char)
                                        (if (eq char #\�)
                                            (subseq string 0 (max 0 (1- (length string))))
                                            (str:concat string (string char))))
                                      line
                                      :initial-value ""))
                            lines))
                   (string #\n ewline))
                  output-stream))
               (decode-process-output-to-stream process output-stream))))
    (decode-process-output-to-stream process t)))

with foo.sh:

#!/bin/sh

printf "backspace test: begin-ERR\b\b\bOR\b\bend\n\bline0"
echo
printf "newline test: line1\nline2"
echo
printf "CRLF newline test: line3\r\nline4"
echo

It would be super useful to integrate this to CMD. I suggest adding a dynamic variable where the user could specify the output translators. When NIL, no translation would happen.

Thoughts?

[Ambrevar]

Update with ^M handled as rubout when not followed by ^J:

(let ((process (uiop:launch-program "foo.sh" :output :stream :error-output :output)))
  (labels ((decode-process-output-to-stream (process output-stream)
             (alex:when-let ((line (read-line (uiop:process-info-output process) nil nil)))
               (setf line (string-right-trim '(#\return) line))
               (write-string
                (sera:ensure-suffix
                 (reduce (lambda (&optional string char)
                           (match char
                             (#\backspace
                                  (subseq string 0 (max 0 (1- (length string)))))
                             (#\return
                                  "")
                             (char (str:concat string (string char)))))
                         line
                         :initial-value "")
                 (string #\newline))
                output-stream)
               (decode-process-output-to-stream process output-stream))))
    (decode-process-output-to-stream process t)))

[ruricolist]

I think that stripping out special characters is a great idea.

Do you think it would be worth having a mode like less - R, that strips everything but color codes?

[Ambrevar]

Sorry, I'm not familiar with `less - R`, what do you mean? Anyways, true that I didn't take coloring into account in this draft, although I'm not completely sure how prevalent it is considering that _well-behaved_ command line programs are supposed to check if they write to a PTY, and if not, disable ANSI coloring. That said, the same could be said about progress bars and all kind of input rewriting, but in practice it's not always done. So... Yes, it's a good idea! There are some CL libs for ANSI coloring out there (like https://github.com/raydeejay/ansi-color), we can use it as a test bed.

[Ambrevar]

Should I send a patch?

[ruricolist]

Please do.

[Ambrevar]

I won't have time to send this before a while, so if anyone feels like giving it a shot, please go ahead :)