How to deal with input strings containing the default escape character \

[mysupply-jlm]

Hi! Thanks for reporting this bug!

Please fill in the relevant information from this template so that we can help you best.

Smart.Format version: 3.5.3

Framework version: net8.0
Microsoft Visual Studio Professional 2022, Version 17.12.4

• Please provide a working source code example to reproduce the bug:

            // This will throw the exception 'Unrecognized escape sequence "\M" in literal.')
            Dictionary<string, string> keyValuePairs = new Dictionary<string, string>();

            string value = @"localhost\MSSQL2019";
            string updated = Smart.Format(value, keyValuePairs);

• What is the current result?
  Two excaped backslash is reduced to one excaped backslash.
  One excaped backslash is seen as a escape charater, and will throw the exception 'Unrecognized escape sequence "\M" in literal.'

• What is the expected result?
  That Smart.Format did not change the value / can handle a text string with a backslash.

• Please post full exception details if applicable (message, stacktrace, inner exceptions)

   at SmartFormat.Core.Parsing.EscapedLiteral.UnEscapeCharLiterals(Char escapingSequenceStart, ReadOnlySpan`1 input, Boolean includeFormatterOptionChars, Span`1 resultBuffer)
   at SmartFormat.Core.Parsing.LiteralText.AsSpan()
   at SmartFormat.Core.Parsing.LiteralText.ToString()
   at SmartFormat.Evaluator.WriteFormat(FormattingInfo formattingInfo)
   at SmartFormat.SmartFormatter.ExecuteFormattingAction(SmartFormatter formatter, IFormatProvider provider, Format formatParsed, IList`1 args, IOutput output, Action`1 doWork)
   at SmartFormat.SmartFormatter.FormatInto(IOutput output, IFormatProvider provider, Format formatParsed, IList`1 args)
   at SmartFormat.SmartFormatter.Format(IFormatProvider provider, Format formatParsed, IList`1 args)
   at SmartFormat.SmartFormatter.Format(IFormatProvider provider, String format, IList`1 args)
   at SmartFormat.SmartFormatter.Format(String format, Object[] args)
   at SmartFormat.Smart.Format(String format, Object arg0)
   at TaskSchedulerServer.UnitTest.TaskScheduler.Utilities.ConfigurationResultUtilTest.ThisWillFail() in ....

• Did you find a workaround? yes/no
  No.

• Is there a version in which it worked?
  Don't know

• Can you help us by writing an unit test?

This illustrate the problem, that two excaped backslash is reduced to one excaped backslash

[Fact]
 public void ValueShouldNotHaveBeenUpdated()
 {            
     IDictionary<string, string> keyValuePairs = new Dictionary<string, string>();

     string value = @"localhost\\MSSQL2019";
     string updated = Smart.Format(value, keyValuePairs);

     Assert.Equal(value, updated);

     /*
     Assert.Equal() Failure: Strings differ
                           ↓ (pos 10)
     Expected: "localhost\\\\MSSQL2019"
     Actual:   "localhost\\MSSQL2019"
                           ↑ (pos 10)
      */
 }       

This demonstrate who Smart.Format, first reduce the two excaped backslash to one, and then see excaped backslash, as a escape charater.

[Fact]
public void ThisWillFail()
{
    try
    {
        IDictionary<string, string> keyValuePairs = new Dictionary<string, string>();
        keyValuePairs["key"] = "value";

        string server = "localhost\\\\MSSQL2019";
        // Server value is 'localhost\\MSSQL2019' - Note the double backslash.

        // This will replace the double backslash, with one backslash
        server = Smart.Format(server, keyValuePairs);
        // Server value is now 'localhost\MSSQL2019' only one backslash

        // If I now try to use the server value, with one backslash,
        // it will throw an exception: 'Unrecognized escape sequence "\M" in literal.'
        server = Smart.Format(server, keyValuePairs);
    }
    catch (Exception ex)
    {
        Assert.Fail(ex.Message);
    }
}

Jacob Mogensen
mySupply ApS
Denmark

[mysupply-jlm]

Is this issue kind of the same problem as Issue 200?

Jacob Mogensen
mySupply ApS
Denmark

[axunonb]

The exception is causes by a disallowed escape sequence -\M in particular.

There are 2 alternatives so resolve this issue, especially when the input string is not under your control:

1. Disable conversion of character string literals:

Smart.Default.Settings.Parser.ConvertCharacterStringLiterals = false;
var result = Smart.Format(@"abc\MSD");
// result is ""abc\MSD"

2. Choose another escape character (it must not interfere with other reserved characters like {}[] and others)

Smart.Default.Settings.Parser.CharLiteralEscapeChar = '^';
var result = Smart.Format(@"abc\MSD");
// result is ""abc\MSD"

See also this section in the Wiki..

Please let us know what would have helped you to resolve the issue by yourself?
Any suggestions are welcome.

[mysupply-jlm]

Hello again,

Thank you for your kind response. What you've suggested as potential fixes does however not appear to actually align with the issue reported.

The reported issue refers to an issue regarding handling of escape characters. The double backslash in the provided examples should be interpreted as an escaped backslash and hence not lead to the illegal "\M" escape sequence.

We see that escape sequences like \n, \t are handled appropriately no matter the amount of sequential escape sequences. For example:

char newLine = '\n';
Assert.Equal("localhost\nMSSQL2019", Smart.Format("localhost\nMSSQL2019", keyValuePairs));

char tap = '\t';
Assert.Equal("localhost\tMSSQL2019", Smart.Format("localhost\tMSSQL2019", keyValuePairs));

Assert.Equal("localhost\t\tMSSQL2019", Smart.Format("localhost\t\tMSSQL2019", keyValuePairs));
Assert.Equal("localhost\t\nMSSQL2019", Smart.Format("localhost\t\nMSSQL2019", keyValuePairs));

All of these result in equally formatted strings. The same is just not the case when escaping backslashes:

char backslas = '\\';
Assert.Equal("localhost\\MSSQL2019", Smart.Format("localhost\\MSSQL2019", keyValuePairs));

... Will result in two non-equal strings because the backslash escape sequence is not properly handled and the exception gets thrown because it thinks the backslash is no longer part of an escape sequence.

A specifically grim example is the following:

string nowContainsANewLineCharater = Smart.Format("localhost\\nMSSQL2019", keyValuePairs);

This will result in a string containing no backslashes at all - instead we simply get a newline character. I would not expect this to be the actual behaviour but please do correct me if there's a limitation to Smart.Format() in this regard.

The general behaviour can be summized to the following:
If Smart.Format() receives a string containing the "\" the number of backslashes gets halfed. No matter the equal amount of backslashes within the character, the returned formatted string will contain half the amount of the input.

[axunonb]

Looks like mixing up regular string literals vs. verbatim string literals in the format string?

Here is a more elaborate explanation than the one from this section in the Wiki..

Regular String Literals

    // Input string contains \n = 0x0A
    Assert.That(Smart.Format("localhost\nMSSQL2019"), Is.EqualTo("localhost\nMSSQL2019"));

    // Input string contains \t = 0x09
    Assert.That(Smart.Format("localhost\tMSSQL2019"), Is.EqualTo("localhost\tMSSQL2019"));

    // Input string contains \t = 0x09
    Assert.That(Smart.Format("localhost\t\tMSSQL2019"), Is.EqualTo("localhost\t\tMSSQL2019"));

    // Input string contains \n as 0x0A, and  \t as 0x09
    Assert.That(Smart.Format("localhost\t\nMSSQL2019"), Is.EqualTo("localhost\t\nMSSQL2019"));

    // Input string contains a single backslash to escape the next character M. This throws here.
    Assert.That(() => Smart.Format("localhost\\MSSQL2019"), Throws.TypeOf<ArgumentException>());

Verbatim String Literals

    // Input string contains \n as a character literal
    Assert.That(Smart.Format(@"localhost\nMSSQL2019"), Is.EqualTo("localhost\nMSSQL2019"));

    // Input string contains \t as a character literal
    Assert.That(Smart.Format(@"localhost\tMSSQL2019"), Is.EqualTo("localhost\tMSSQL2019"));

    // Input string contains \t as a character literal
    Assert.That(Smart.Format(@"localhost\t\tMSSQL2019"), Is.EqualTo("localhost\t\tMSSQL2019"));

    // Input string contains \n\t as a character literal
    Assert.That(Smart.Format(@"localhost\t\nMSSQL2019"), Is.EqualTo("localhost\t\nMSSQL2019"));

    // Input string contains a single backslash to escape the next character \. This works here.
    Assert.That(Smart.Format(@"localhost\\MSSQL2019"), Is.EqualTo("localhost\\MSSQL2019"));

[axunonb]

@mysupply-jlm Did this explanation help to understand the difference, and how Smart.Format uses string literals?

[mysupply-jlm]

Hey @axunonb

Actually, it did help me understand why SmartFormat behave like it does.
However, I do think it is wrong, and a none expected behavior. Try to follow my logic here:

The verbatime string

string value = @"localhost\MSSQL2019"

is actually the string

string value = "localhost\\MSSQL2019"

and

string value = @"localhost\\MSSQL2019"

is actually the string

string value = "localhost\\\\MSSQL2019"

The string value is localhost\MSSQL2019, and is written in C# a localhost\\MSSQL2019, where the first backslash, is an escape charater.

When that valid string is formatted by SmartFormat, it will result in the error. SmartFormat by default, tries to fix a 'wrong' input (verbatime) string, and replace two backslashes with one backslash, and thereby resulting in an error.

I have nothing against that feature. I do like it and can see its value. My problem is that it is enabled by default.

If users of SmartFormat use verbatim string, and thereby sending a ''wrong'' string to SmartFormat, and the users want SmartFormat to fix it, they should enable that feature. It should not be the other way around.

Correct use of string values should NOT be the one to disable a feature.