Support runtest.py to work on Windows when no_pty=True

[cy20lin]

Tweak runtest.py script so that it could now partially run on Windows. The pty part is not implemented, and would raise if --no-pty is not specified.

For the implementation, the threading and queue modules are used to read the subprocess.stdout pipe with timeout, as the select module doesn't work on Windows.

[OldLiu001]

I briefly tested it and it seemed to add some strange token at the end of each line, like \x16 or something.

This caused both my VBScript implementation and the repository's original PowerShell implementation by Joel Martin to fail the test (although it produced the expected results).

[cy20lin]

Thanks for your feedback. The special character \x16 was introduced at commit 18616b1 by @kanaka.

As I have no clear Idea about how the string '\x16\r' work at Runner.writeline on posix, I keep that part intact on posix platform and only modify the code to write the normal str with line_break to the stdin of the subprocess on Windows.

I've tested the powershell implementation by Joel Martin from step0 to step9, and the tests passed smoothly. Except for the step5, you might wants to increase --test-timeout to avoid timeout when testing (def! res2 (sum2 10000 0)).

Regarding stepA, there is an hanging issue when testing (readline "mal-user> "), the same issue is also found when testing on Windows WSL Ubuntu. I'm now fixing that issue and will update if I make further progress.

[OldLiu001]

Thanks for update.

But when I test step0 with PowerShell impl, got this:

Testing basic string
] -> FAIL (line 3): -> ['',abcABC123
    Expected : 'abcABC123\\\r'
    Got      : 'abcABC123'
Testing string containing spaces
] -> FAIL (line 7):rld\r' -> ['',hello mal world
    Expected : 'hello\\ mal\\ world\\\r'
    Got      : 'hello mal world'
Testing string containing symbols
] -> FAIL (line 11):\r' -> ['',[]{}"'* ;:()
    Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
    Got      : '[]{}"\'* ;:()'
Test long string
TEST: 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r' -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'] -> FAIL (line 16):
    Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
Non alphanumeric characters
] -> FAIL (line 20):
    Expected : '!\\\r'
    Got      : '!'
] -> FAIL (line 22):
    Expected : '\\&\\\r'
    Got      : '&'
] -> FAIL (line 24):
    Expected : '\\+\\\r'
    Got      : '+'
] -> FAIL (line 26):
    Expected : ',\\\r'
    Got      : ','
] -> FAIL (line 28):
    Expected : '\\-\\\r'
    Got      : '-'
] -> FAIL (line 30):
    Expected : '/\\\r'
    Got      : '/'
] -> FAIL (line 32):
    Expected : '<\\\r'
    Got      : '<'
] -> FAIL (line 34):
    Expected : '=\\\r'
    Got      : '='
] -> FAIL (line 36):
    Expected : '>\\\r'
    Got      : '>'
] -> FAIL (line 38):
    Expected : '\\?\\\r'
    Got      : '?'
] -> FAIL (line 40):
    Expected : '@\\\r'
    Got      : '@'
] -> FAIL (line 43):
    Expected : '\\^\\\r'
    Got      : '^'
] -> FAIL (line 45):
    Expected : '_\\\r'
    Got      : '_'
] -> FAIL (line 47):
    Expected : '`\\\r'
    Got      : '`'
] -> FAIL (line 49):
    Expected : '\\~\\\r'
    Got      : '~'
------- Optional Functionality --------------
------- (Not needed for self-hosting) -------
Non alphanumeric characters
] -> SOFT FAIL (line 58):
    Expected : '\\#\\\r'
    Got      : '#'
] -> SOFT FAIL (line 60):
    Expected : '\\$\\\r'
    Got      : '$'
] -> SOFT FAIL (line 62):
    Expected : '%\\\r'
    Got      : '%'
] -> SOFT FAIL (line 64):
    Expected : '\\.\\\r'
    Got      : '.'
] -> SOFT FAIL (line 66):
    Expected : '\\|\\\r'
    Got      : '|'

FAILURES:
]:> ['',abcABC123 3): abcABC123
    Expected : 'abcABC123\\\r'
    Got      : 'abcABC123'
]:> ['',hello mal worldello mal world
    Expected : 'hello\\ mal\\ world\\\r'
    Got      : 'hello mal world'
]:> ['',[]{}"'* ;:()): []{}"'* ;:()
    Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
    Got      : '[]{}"\'* ;:()'
FAILED TEST (line 16): hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() [ -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{]:'*)
    Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
]:> ['',!ST (line 20): !
    Expected : '!\\\r'
    Got      : '!'
]:> ['',&ST (line 22): &
    Expected : '\\&\\\r'
    Got      : '&'
]:> ['',+ST (line 24): +
    Expected : '\\+\\\r'
    Got      : '+'
]:> ['',,ST (line 26): ,
    Expected : ',\\\r'
    Got      : ','
]:> ['',-ST (line 28): -
    Expected : '\\-\\\r'
    Got      : '-'
]:> ['',/ST (line 30): /
    Expected : '/\\\r'
    Got      : '/'
]:> ['',<ST (line 32): <
    Expected : '<\\\r'
    Got      : '<'
]:> ['',=ST (line 34): =
    Expected : '=\\\r'
    Got      : '='
]:> ['',>ST (line 36): >
    Expected : '>\\\r'
    Got      : '>'
]:> ['',?ST (line 38): ?
    Expected : '\\?\\\r'
    Got      : '?'
]:> ['',@ST (line 40): @
    Expected : '@\\\r'
    Got      : '@'
]:> ['',^ST (line 43): ^
    Expected : '\\^\\\r'
    Got      : '^'
]:> ['',_ST (line 45): _
    Expected : '_\\\r'
    Got      : '_'
]:> ['',`ST (line 47): `
    Expected : '`\\\r'
    Got      : '`'
]:> ['',~ST (line 49): ~
    Expected : '\\~\\\r'
    Got      : '~'
]:> ['',#ED TEST (line 58): #
    Expected : '\\#\\\r'
    Got      : '#'
]:> ['',$ED TEST (line 60): $
    Expected : '\\$\\\r'
    Got      : '$'
]:> ['',%ED TEST (line 62): %
    Expected : '%\\\r'
    Got      : '%'
]:> ['',.ED TEST (line 64): .
    Expected : '\\.\\\r'
    Got      : '.'
]:> ['',|ED TEST (line 66): |
    Expected : '\\|\\\r'
    Got      : '|'

TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step0_repl.mal):
    5: soft failing tests
   19: failing tests
    0: passing tests
   24: total tests

I fix it:

elif os.name == 'nt':
        expects = ["%s%s" % (t.out, re.escape(t.ret))]

with this:

elif os.name == 'nt':
        expects = ["%s%s" % (t.out, re.escape(t.ret)),
                   "%s%s" % (t.out, re.escape(t.ret.strip()))]

Then step0 of powershell & vbs can pass smoothly.

And when I test step1,

With powershell I got these 'FAILURES':

FAILURES:
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ')', got EOF"
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ']', got EOF"
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ')', got EOF"
 -> ['{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r',]:
    Expected : '{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r'
    Got      : '{"a2" 2 "a1" 1 "a3" 3}'

TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step1_read_print.mal):
    0: soft failing tests
    9: failing tests
  108: passing tests
  117: total tests

With my vbs impl, It stuck in ;; whole line comment (not an exception)\r,

] -> SUCCESS  {:b   {  :cde     3   }  }}\r' -> ['',{:a {:b {:cde 3}}}
] -> SUCCESS1}\r' -> ['',{"1" 1}
] -> SUCCESSr' -> ['',({})
Testing read of comments
TEST: ' ;; whole line comment (not an exception)\r' -> ['',] -> TIMEOUT (line 226)

Exception: TestTimeout('TIMEOUT (line 226)')
Output before exception:
user>

So I test my vbs's step1 with pipe:

mal\impls\vbs> .\run_step1_read_print.cmd < ..\tests\step1_read_print.mal

It did not get stuck and generate the expected result.

user> user> 1
user> user> 7
user> user> 7
user> user> -123
user> user> user> user> user> +
user> user> abc
user> user> abc
user> user> abc5
user> user> abc-def
user> user> user> user> -
user> user> -abc
user> user> ->>
user> user> user> user> (+ 1 2)
user> user> ()
user> user> ()
user> user> (nil)
user> user> ((3 4))
user> user> (+ 1 (+ 2 3))
user> user> (+ 1 (+ 2 3))
user> user> (* 1 2)
user> user> (** 1 2)
user> user> (* -3 6)
user> user> (() ())
user> user> user> user> (1 2 3)
user> user> user> user> user> user> user> user> user> user> nil
user> user> true
user> user> false
user> user> user> user> "abc"
user> user> "abc"
user> user> "abc (with parens)"
user> user> "abc\"def"
user> user> ""
user> user> "\\"
user> user> "\\\\\\\\\\\\\\\\\\"
user> user> "&"
user> user> "'"
user> user> "("
user> user> ")"
user> user> "*"
user> user> "+"
user> user> ","
user> user> "-"
user> user> "/"
user> user> ":"
user> user> ";"
user> user> "<"
user> user> "="
user> user> ">"
user> user> "?"
user> user> "@"
user> user> "["
user> user> "]"
user> user> "^"
user> user> "_"
user> user> "`"
user> user> "{"
user> user> "}"
user> user> "~"
user> user> "!"
user> user> user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> (quote 1)
user> user> (quote (1 2 3))
user> user> (quasiquote 1)
user> user> (quasiquote (1 2 3))
user> user> (unquote 1)
user> user> (unquote (1 2 3))
user> user> (quasiquote (1 (unquote a) 3))
user> user> (splice-unquote (1 2 3))
user> user> user> user> user> :kw
user> user> (:kw1 :kw2 :kw3)
user> user> user> user> [+ 1 2]
user> user> []
user> user> []
user> user> [[3 4]]
user> user> [+ 1 [+ 2 3]]
user> user> [+ 1 [+ 2 3]]
user> user> ([])
user> user> user> user> {}
user> user> {}
user> user> {"abc" 1}
user> user> {"a" {"b" 2}}
user> user> {"a" {"b" {"c" 3}}}
user> user> {"a" {"b" {"cde" 3}}}
user> user> user> user> {"a1" 1 "a2" 2 "a3" 3}
user> user> {:a {:b {:cde 3}}}
user> user> {"1" 1}
user> user> ({})
user> user> user> user> user> 1
user> user> 1
user> user> user> user> (deref a)
user> user> user> user> user> user> user> user> user> (with-meta [1 2 3] {"a" 1})
user> user> user> user> user> user> "\n"
user> user> "#"
user> user> "$"
user> user> "%"
user> user> "."
user> user> "\\"
user> user> "|"
user> user> user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> user> 1
user> user>

And then I manually test all tests in step1_read_print.mal with my vbs impl, It also not get stuck.

Here is my test script (Batch File, placed in same folder with runtest.py):

@echo off
pushd "%~dp0"
goto p
for %%a in (
	step0_repl
	step1_read_print
) do (
	echo @pushd "%%~dp0" ^& @cscript -nologo %%a.vbs > .\impls\vbs\run_%%a.cmd
	python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\vbs\run_%%a.cmd"
	pause
)
exit
:p
for %%a in (
	step0_repl
	step1_read_print
) do (
	echo @pushd "%%~dp0" ^& @powershell ".\%%a.ps1" > .\impls\powershell\run_%%a.cmd
	python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\powershell\run_%%a.cmd"
	pause
)

May I ask what your PowerShell testing environment is? I think there may be slight differences in our environment, which may result in your PowerShell testing passing while mine cannot.

[cy20lin]

I fix the hanging issue when testing on stepA. I test from step0 to stepA again for all my environments and the runtest.py script runs smoomthly.

I have two testing environments, which are:

• First
  • Windows 11
  • Python 3.11.4
  • PSVersion 5.1.22621.1778
• Second
  • Ubuntu 22.04 WSL on Windows 11
  • Python 3.10.6
  • PSVersion 7.3.5

To test in my first environment, I use the following script.

@echo off
@REM test.cmd
@REM Put this script in the same directory as the runtest.py file

@REM # How to run
@REM # O. Start In powershell shell environment
@REM # 1. Change to Mal project directory 
@REM cd path\to\mal
@REM # 2. Run the test and log the result
@REM .\test.cmd | Tee-Object -FilePath test-nt.log

for %%a in (
    step0_repl
    step1_read_print
    step2_eval
    step3_env
    step4_if_fn_do
    step5_tco
    step6_file
    step7_quote
    step8_macros
    step9_try
    stepA_mal
) do (
	python runtest.py --rundir "impls\powershell" --test-timeout 60 --deferrable --optional --no-pty "..\tests\%%a.mal" powershell ".\%%a.ps1"
)

To test in my second environment, I use the following script.

#!/usr/bin/bash
# test.bash

# Put this script in the same directory as the runtest.py file

# How to run
# # 1. Change to Mal project directory 
# cd path/to/mal
# # 2. Run the test and log the result
# bash test.bash | tee test-posix.log

names=(
    step0_repl
    step1_read_print
    step2_eval
    step3_env
    step4_if_fn_do
    step5_tco
    step6_file
    step7_quote
    step8_macros
    step9_try
    stepA_mal
)

for name in "${names[@]}"
do
    python runtest.py --rundir "impls/powershell" --test-timeout 60 --deferrable --optional --no-pty "../tests/${name}.mal" pwsh "${name}.ps1"
done

One could refer to the attached files for the test log.

• test-nt.log
• test-posix.log

[cy20lin]

@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.

[OldLiu001]

@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.

I finally discovered the root cause of the problem.

In Windows, git defaults to changing the line feed encoding of the cloned text file to CRLF, while runtest.py only considers the case where the line feed encoding is LF (i.e. split('\n')).

Resulting in an additional \r at the end of the generated regular expression and leading to test failure.