'main': Highlight backslash escape sequences outside quotes.

[danielshahaf]

Fixes #631.

WIP: test expectations should be updated (12 failures including cthulhu)

[danielshahaf]

@phy1729 Sanity check please before I update test expectations?

«make quiet-test» output

## assign-not-array2
# BUFFER=$'a=foo\\( :'
ok 1 - [1,7] «a=foo\(»
ok 2 - [3,7] «foo\(»
not ok 3 - [6,7] «\(» - expected (9 9 "builtin"), observed (6 7 "back-un-quoted-argument"). 
not ok 4 - cardinality check - have 3 expectations and 4 region_highlight entries: «expected_region_highlight=( $'1 7 assign' $'3 7 default' $'9 9 builtin' )» «region_highlight=( $'0 7 assign' $'2 7 default' $'5 7 back-un-quoted-argument' $'8 9 builtin' )»
# expected_region_highlight  region_highlight
# '1 7 assign'               '0 7 assign'
# '3 7 default'              '2 7 default'
# '9 9 builtin'              '5 7 back-un-quoted-argument'
# .                          '8 9 builtin'

## backslash-space
# BUFFER=$'echo \\ \'foo\' ; ls'
ok 1 - [1,4] «echo»
ok 2 - [6,12] «\ 'foo'»
not ok 3 - [6,7] «\ » - expected (8 12 "single-quoted-argument"), observed (6 7 "back-un-quoted-argument"). 
not ok 4 - [8,12] «'foo'» - expected (14 14 "commandseparator"), observed (8 12 "single-quoted-argument"). 
not ok 5 - [14,14] «;» - expected (16 17 "command"), observed (14 14 "commandseparator"). 
not ok 6 - cardinality check - have 5 expectations and 6 region_highlight entries: «expected_region_highlight=( $'1 4 builtin' $'6 12 default' $'8 12 single-quoted-argument' $'14 14 commandseparator' $'16 17 command' )» «region_highlight=( $'0 4 builtin' $'5 12 default' $'5 7 back-un-quoted-argument' $'7 12 single-quoted-argument' $'13 14 commandseparator' $'15 17 command' )»
# expected_region_highlight      region_highlight
# '1 4 builtin'                  '0 4 builtin'
# '6 12 default'                 '5 12 default'
# '8 12 single-quoted-argument'  '5 7 back-un-quoted-argument'
# '14 14 commandseparator'       '7 12 single-quoted-argument'
# '16 17 command'                '13 14 commandseparator'
# .                              '15 17 command'

## cthulhu
# BUFFER=$'echo Ph\\\'ng`echo lui "mg"\\`echo lw\\\'nafh \\\\\\`echo Cthu"lhu\\\\\\` R\\\\\'ly$(echo eh wag\\\\\\`echo h\\\'nag\\\\\\`\'l\' fht)agn`'
ok 1 - [1,4] «echo»
ok 2 - [6,113] «Ph\'ng`echo lui "mg"\`echo lw\'nafh \\\`echo Cthu"lhu\\\` R\\'ly$(echo eh wag\\\`echo h\'nag\\\`'l' fht)agn`»
not ok 3 - [8,9] «\'» - expected (12 113 "back-quoted-argument"), observed (8 9 "back-un-quoted-argument"). 
not ok 4 - [12,113] «`echo lui "mg"\`echo lw\'nafh \\\`echo Cthu"lhu\\\` R\\'ly$(echo eh wag\\\`echo h\'nag\\\`'l' fht)agn`» - expected (12 12 "back-quoted-argument-delimiter"), observed (12 113 "back-quoted-argument"). 
not ok 5 - [12,12] «`» - expected (13 16 "builtin"), observed (12 12 "back-quoted-argument-delimiter"). 
not ok 6 - [13,16] «echo» - expected (18 20 "default"), observed (13 16 "builtin"). 
not ok 7 - [18,20] «lui» - expected (22 112 "default"), observed (18 20 "default"). 
not ok 8 - [22,112] «"mg"\`echo lw\'nafh \\\`echo Cthu"lhu\\\` R\\'ly$(echo eh wag\\\`echo h\'nag\\\`'l' fht)agn» - expected (22 25 "double-quoted-argument"), observed (22 112 "default"). 
not ok 9 - [22,25] «"mg"» - expected (26 112 "back-quoted-argument-unclosed"), observed (22 25 "double-quoted-argument"). 
not ok 10 - [26,112] «\`echo lw\'nafh \\\`echo Cthu"lhu\\\` R\\'ly$(echo eh wag\\\`echo h\'nag\\\`'l' fht)agn» - expected (26 27 "back-quoted-argument-delimiter"), observed (26 112 "back-quoted-argument-unclosed"). 
not ok 11 - [26,27] «\`» - expected (28 31 "builtin"), observed (26 27 "back-quoted-argument-delimiter"). 
not ok 12 - [28,31] «echo» - expected (33 40 "default"), observed (28 31 "builtin"). 
not ok 13 - [33,40] «lw\'nafh» - expected (42 62 "default"), observed (33 40 "default"). 
not ok 14 - [35,36] «\'» - expected (42 62 "back-quoted-argument"), observed (35 36 "back-un-quoted-argument"). 
not ok 15 - [42,62] «\\\`echo Cthu"lhu\\\`» - expected (42 45 "back-quoted-argument-delimiter"), observed (42 62 "default"). 
not ok 16 - [42,62] «\\\`echo Cthu"lhu\\\`» - expected (46 49 "builtin"), observed (42 62 "back-quoted-argument"). 
not ok 17 - [42,45] «\\\`» - expected (51 58 "default"), observed (42 45 "back-quoted-argument-delimiter"). 
not ok 18 - [46,49] «echo» - expected (55 58 "double-quoted-argument-unclosed"), observed (46 49 "builtin"). 
not ok 19 - [51,58] «Cthu"lhu» - expected (59 62 "back-quoted-argument-delimiter"), observed (51 58 "default"). 
not ok 20 - [55,58] «"lhu» - expected (64 112 "default"), observed (55 58 "double-quoted-argument-unclosed"). 
not ok 21 - [59,62] «\\\`» - expected (70 109 "command-substitution-unquoted"), observed (59 62 "back-quoted-argument-delimiter"). 
not ok 22 - [64,112] «R\\'ly$(echo eh wag\\\`echo h\'nag\\\`'l' fht)agn» - expected (70 71 "command-substitution-delimiter-unquoted"), observed (64 112 "default"). 
not ok 23 - [65,67] «\\'» - expected (72 75 "builtin"), observed (65 67 "back-un-quoted-argument"). 
not ok 24 - [70,109] «$(echo eh wag\\\`echo h\'nag\\\`'l' fht)» - expected (77 78 "default"), observed (70 109 "command-substitution-unquoted"). 
not ok 25 - [70,71] «$(» - expected (80 104 "default"), observed (70 71 "command-substitution-delimiter-unquoted"). 
not ok 26 - [72,75] «echo» - expected (83 101 "back-quoted-argument"), observed (72 75 "builtin"). 
not ok 27 - [77,78] «eh» - expected (83 86 "back-quoted-argument-delimiter"), observed (77 78 "default"). 
not ok 28 - [80,104] «wag\\\`echo h\'nag\\\`'l'» - expected (87 90 "builtin"), observed (80 104 "default"). 
not ok 29 - [83,101] «\\\`echo h\'nag\\\`» - expected (92 97 "default"), observed (83 101 "back-quoted-argument"). 
not ok 30 - [83,86] «\\\`» - expected (98 101 "back-quoted-argument-delimiter"), observed (83 86 "back-quoted-argument-delimiter"). 
not ok 31 - [87,90] «echo» - expected (102 104 "single-quoted-argument"), observed (87 90 "builtin"). 
not ok 32 - [92,97] «h\'nag» - expected (106 108 "default"), observed (92 97 "default"). 
not ok 33 - [93,94] «\'» - expected (109 109 "command-substitution-delimiter-unquoted"), observed (93 94 "back-un-quoted-argument"). 
not ok 34 - [98,101] «\\\`» - expected (113 113 "unknown-token"), observed (98 101 "back-quoted-argument-delimiter"). 
not ok 35 - cardinality check - have 34 expectations and 38 region_highlight entries: «expected_region_highlight=( $'1 4 builtin' $'6 113 default' $'12 113 back-quoted-argument' $'12 12 back-quoted-argument-delimiter' $'13 16 builtin' $'18 20 default' $'22 112 default' $'22 25 double-quoted-argument' $'26 112 back-quoted-argument-unclosed' $'26 27 back-quoted-argument-delimiter' $'28 31 builtin' $'33 40 default' $'42 62 default' $'42 62 back-quoted-argument' $'42 45 back-quoted-argument-delimiter' $'46 49 builtin' $'51 58 default' $'55 58 double-quoted-argument-unclosed' $'59 62 back-quoted-argument-delimiter' $'64 112 default' $'70 109 command-substitution-unquoted' $'70 71 command-substitution-delimiter-unquoted' $'72 75 builtin' $'77 78 default' $'80 104 default' $'83 101 back-quoted-argument' $'83 86 back-quoted-argument-delimiter' $'87 90 builtin' $'92 97 default' $'98 101 back-quoted-argument-delimiter' $'102 104 single-quoted-argument' $'106 108 default' $'109 109 command-substitution-delimiter-unquoted' $'113 113 unknown-token' )» «region_highlight=( $'0 4 builtin' $'5 113 default' $'7 9 back-un-quoted-argument' $'11 113 back-quoted-argument' $'11 12 back-quoted-argument-delimiter' $'12 16 builtin' $'17 20 default' $'21 112 default' $'21 25 double-quoted-argument' $'25 112 back-quoted-argument-unclosed' $'25 27 back-quoted-argument-delimiter' $'27 31 builtin' $'32 40 default' $'34 36 back-un-quoted-argument' $'41 62 default' $'41 62 back-quoted-argument' $'41 45 back-quoted-argument-delimiter' $'45 49 builtin' $'50 58 default' $'54 58 double-quoted-argument-unclosed' $'58 62 back-quoted-argument-delimiter' $'63 112 default' $'64 67 back-un-quoted-argument' $'69 109 command-substitution-unquoted' $'69 71 command-substitution-delimiter-unquoted' $'71 75 builtin' $'76 78 default' $'79 104 default' $'82 101 back-quoted-argument' $'82 86 back-quoted-argument-delimiter' $'86 90 builtin' $'91 97 default' $'92 94 back-un-quoted-argument' $'97 101 back-quoted-argument-delimiter' $'101 104 single-quoted-argument' $'105 108 default' $'108 109 command-substitution-delimiter-unquoted' $'112 113 unknown-token' )»
# expected_region_highlight                          region_highlight
# '1 4 builtin'                                      '0 4 builtin'
# '6 113 default'                                    '5 113 default'
# '12 113 back-quoted-argument'                      '7 9 back-un-quoted-argument'
# '12 12 back-quoted-argument-delimiter'             '11 113 back-quoted-argument'
# '13 16 builtin'                                    '11 12 back-quoted-argument-delimiter'
# '18 20 default'                                    '12 16 builtin'
# '22 112 default'                                   '17 20 default'
# '22 25 double-quoted-argument'                     '21 112 default'
# '26 112 back-quoted-argument-unclosed'             '21 25 double-quoted-argument'
# '26 27 back-quoted-argument-delimiter'             '25 112 back-quoted-argument-unclosed'
# '28 31 builtin'                                    '25 27 back-quoted-argument-delimiter'
# '33 40 default'                                    '27 31 builtin'
# '42 62 default'                                    '32 40 default'
# '42 62 back-quoted-argument'                       '34 36 back-un-quoted-argument'
# '42 45 back-quoted-argument-delimiter'             '41 62 default'
# '46 49 builtin'                                    '41 62 back-quoted-argument'
# '51 58 default'                                    '41 45 back-quoted-argument-delimiter'
# '55 58 double-quoted-argument-unclosed'            '45 49 builtin'
# '59 62 back-quoted-argument-delimiter'             '50 58 default'
# '64 112 default'                                   '54 58 double-quoted-argument-unclosed'
# '70 109 command-substitution-unquoted'             '58 62 back-quoted-argument-delimiter'
# '70 71 command-substitution-delimiter-unquoted'    '63 112 default'
# '72 75 builtin'                                    '64 67 back-un-quoted-argument'
# '77 78 default'                                    '69 109 command-substitution-unquoted'
# '80 104 default'                                   '69 71 command-substitution-delimiter-unquoted'
# '83 101 back-quoted-argument'                      '71 75 builtin'
# '83 86 back-quoted-argument-delimiter'             '76 78 default'
# '87 90 builtin'                                    '79 104 default'
# '92 97 default'                                    '82 101 back-quoted-argument'
# '98 101 back-quoted-argument-delimiter'            '82 86 back-quoted-argument-delimiter'
# '102 104 single-quoted-argument'                   '86 90 builtin'
# '106 108 default'                                  '91 97 default'
# '109 109 command-substitution-delimiter-unquoted'  '92 94 back-un-quoted-argument'
# '113 113 unknown-token'                            '97 101 back-quoted-argument-delimiter'
# .                                                  '101 104 single-quoted-argument'
# .                                                  '105 108 default'
# .                                                  '108 109 command-substitution-delimiter-unquoted'
# .                                                  '112 113 unknown-token'

## escaped-single-quote
# BUFFER=$': \'foo\'\\\'\'bar\''
ok 1 - [1,1] «:»
ok 2 - [3,14] «'foo'\''bar'»
ok 3 - [3,7] «'foo'»
not ok 4 - [8,9] «\'» - expected (10 14 "single-quoted-argument"), observed (8 9 "back-un-quoted-argument"). 
not ok 5 - cardinality check - have 4 expectations and 5 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 14 default' $'3 7 single-quoted-argument' $'10 14 single-quoted-argument' )» «region_highlight=( $'0 1 builtin' $'2 14 default' $'2 7 single-quoted-argument' $'7 9 back-un-quoted-argument' $'9 14 single-quoted-argument' )»
# expected_region_highlight       region_highlight
# '1 1 builtin'                   '0 1 builtin'
# '3 14 default'                  '2 14 default'
# '3 7 single-quoted-argument'    '2 7 single-quoted-argument'
# '10 14 single-quoted-argument'  '7 9 back-un-quoted-argument'
# .                               '9 14 single-quoted-argument'

## glob
# BUFFER=$': foo* bar? *baz qux\\?'
ok 1 - [1,1] «:»
ok 2 - [3,6] «foo*»
ok 3 - [6,6] «*»
ok 4 - [8,11] «bar?»
ok 5 - [11,11] «?»
ok 6 - [13,16] «*baz»
ok 7 - [13,13] «*»
ok 8 - [18,22] «qux\?»
not ok 9 - cardinality check - have 8 expectations and 9 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 6 default' $'6 6 globbing' $'8 11 default' $'11 11 globbing' $'13 16 default' $'13 13 globbing' $'18 22 default' )» «region_highlight=( $'0 1 builtin' $'2 6 default' $'5 6 globbing' $'7 11 default' $'10 11 globbing' $'12 16 default' $'12 13 globbing' $'17 22 default' $'20 22 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 1 builtin'              '0 1 builtin'
# '3 6 default'              '2 6 default'
# '6 6 globbing'             '5 6 globbing'
# '8 11 default'             '7 11 default'
# '11 11 globbing'           '10 11 globbing'
# '13 16 default'            '12 16 default'
# '13 13 globbing'           '12 13 globbing'
# '18 22 default'            '17 22 default'
# .                          '20 22 back-un-quoted-argument'

## order-path-after-dollar
# BUFFER=$': $foo \\$foo'
ok 1 - [1,1] «:»
ok 2 - [3,6] «$foo»
ok 3 - [8,12] «\$foo»
not ok 4 - cardinality check - have 3 expectations and 4 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 6 default' $'8 12 path' )» «region_highlight=( $'0 1 builtin' $'2 6 default' $'7 12 path' $'7 9 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 1 builtin'              '0 1 builtin'
# '3 6 default'              '2 6 default'
# '8 12 path'                '7 12 path'
# .                          '7 9 back-un-quoted-argument'

## order-path-before-globbing
# BUFFER=$': * \\*'
ok 1 - [1,1] «:»
ok 2 - [3,3] «*»
ok 3 - [3,3] «*»
ok 4 - [5,6] «\*»
not ok 5 - cardinality check - have 4 expectations and 5 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 3 default' $'3 3 globbing' $'5 6 path' )» «region_highlight=( $'0 1 builtin' $'2 3 default' $'2 3 globbing' $'4 6 path' $'4 6 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 1 builtin'              '0 1 builtin'
# '3 3 default'              '2 3 default'
# '3 3 globbing'             '2 3 globbing'
# '5 6 path'                 '4 6 path'
# .                          '4 6 back-un-quoted-argument'

## path-dollared-word2
# BUFFER=$': \\$lambda'
ok 1 - [1,1] «:»
ok 2 - [3,10] «\$lambda»
not ok 3 - cardinality check - have 2 expectations and 3 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 10 path' )» «region_highlight=( $'0 1 builtin' $'2 10 path' $'2 4 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 1 builtin'              '0 1 builtin'
# '3 10 path'                '2 10 path'
# .                          '2 4 back-un-quoted-argument'

## path-space
# BUFFER=$'ls A/mu\\ with\\ spaces'
ok 1 - [1,2] «ls»
ok 2 - [4,21] «A/mu\ with\ spaces»
not ok 3 - cardinality check - have 2 expectations and 4 region_highlight entries: «expected_region_highlight=( $'1 2  command' $'4 21 path' )» «region_highlight=( $'0 2 command' $'3 21 path' $'7 9 back-un-quoted-argument' $'13 15 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 2  command'             '0 2 command'
# '4 21 path'                '3 21 path'
# .                          '7 9 back-un-quoted-argument'
# .                          '13 15 back-un-quoted-argument'

## path-tilde-home3
# BUFFER=$'ls \\~'
ok 1 - [1,2] «ls»
ok 2 - [4,5] «\~»
not ok 3 - cardinality check - have 2 expectations and 3 region_highlight entries: «expected_region_highlight=( $'1 2 command' $'4 5 default' )» «region_highlight=( $'0 2 command' $'3 5 default' $'3 5 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 2 command'              '0 2 command'
# '4 5 default'              '3 5 default'
# .                          '3 5 back-un-quoted-argument'

## path_prefix3
# PREBUFFER=$'ls \\'
# BUFFER=$'/bin/s'
ok 1 - [1,6] «/bin/s»
not ok 2 - cardinality check - have 1 expectations and 2 region_highlight entries: «expected_region_highlight=( $'1 6 path_prefix' )» «region_highlight=( $'0 6 path_prefix' $'0 1 back-un-quoted-argument' )»
# expected_region_highlight  region_highlight
# '1 6 path_prefix'          '0 6 path_prefix'
# .                          '0 1 back-un-quoted-argument'

## redirection3
# BUFFER=$': >>>; : <>\\<<<<EOF'
ok 1 - [1,1] «:»
ok 2 - [3,4] «>>»
ok 3 - [5,5] «>»
ok 4 - [6,6] «;»
ok 5 - [8,8] «:»
ok 6 - [10,11] «<>»
ok 7 - [12,13] «\<»
not ok 8 - [12,13] «\<» - expected (14 16 "redirection"), observed (12 13 "back-un-quoted-argument"). 
not ok 9 - [14,16] «<<<» - expected (17 19 "default"), observed (14 16 "redirection"). 
not ok 10 - cardinality check - have 9 expectations and 10 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 4 redirection' $'5 5 unknown-token' $'6 6 commandseparator' $'8 8 builtin' $'10 11 redirection' $'12 13 default' $'14 16 redirection' $'17 19 default' )» «region_highlight=( $'0 1 builtin' $'2 4 redirection' $'4 5 unknown-token' $'5 6 commandseparator' $'7 8 builtin' $'9 11 redirection' $'11 13 default' $'11 13 back-un-quoted-argument' $'13 16 redirection' $'16 19 default' )»
# expected_region_highlight  region_highlight
# '1 1 builtin'              '0 1 builtin'
# '3 4 redirection'          '2 4 redirection'
# '5 5 unknown-token'        '4 5 unknown-token'
# '6 6 commandseparator'     '5 6 commandseparator'
# '8 8 builtin'              '7 8 builtin'
# '10 11 redirection'        '9 11 redirection'
# '12 13 default'            '11 13 default'
# '14 16 redirection'        '11 13 back-un-quoted-argument'
# '17 19 default'            '13 16 redirection'
# .                          '16 19 default'

[phy1729]

path_prefix3 '0 1 back-un-quoted-argument' seems incorrect since it's the newline that's escaped not the first character in BUFFER.

I'm almost regretting Cthulhu exists now, but I'm glad it verified it works when nested in side backticks with '64 67 back-un-quoted-argument'.

[danielshahaf]

Thanks for the review.

path_prefix3 is wrong, isn't it? Can $PREBUFFER ever not end with a $'\n'? When I add a $'\n' to $PREBUFFER in path_prefix3.zsh, the test passes with this branch (didn't check without).

I'll have to verify cthulhu manually, I guess…

[danielshahaf]

path_prefix3 is wrong, isn't it? Can $PREBUFFER ever not end with a $'\n'?

And if that's not it, I'll next check whether it's a bug in this PR or a side-effect of #705 that's exposed by this PR.

[phy1729]

I checked the diff for cthulhu. I was just noting that I'm glad the triple backslash-un-quoted-argument was highlighted in entirety as it should be.

From a quick test typing the PREBUFFER and BUFFER of path_prefix3 and dumping PREBUFFER and BUFFER, I concur the test is wrong and is missing a $'\n' at the end of PREBUFFER.

[danielshahaf]

I checked the diff for cthulhu.

Thanks!

I was just noting that I'm glad the triple backslash-un-quoted-argument was highlighted in entirety as it should be.

*nod* Understood.

From a quick test typing the PREBUFFER and BUFFER of path_prefix3 and dumping PREBUFFER and BUFFER, I concur the test is wrong and is missing a `$'\n'` at the end of PREBUFFER.

Will fix.

[danielshahaf]

Will fix.

Done in f6471db.

[phy1729]

Stray space after "\\"), but otherwise looks good to merge.