Escape sequences in string parsing

[mmatera]

Description

Looking at #1619, I found a slight difference on how WMA and Mathics parse strings with escaped characters.

How to Reproduce

In CLI:

In[1]:="\\\` " //InputForm

Output Given

In Mathics3

Out[1]:="\\\\` " //InputForm

Expected behavior

In WMA,

Out[1]:="\\\` " //InputForm

The issue manifests when using StringForm, because we cannot put a backslash before a placeholder:

In[1]:=StringForm["C:\\`1`", "root"]

In WMA, we get

Out[1]:= C:\root

but in Mathics we get an error:

StringForm::sfq: Unmatched backquote in C:\`1`.
Out[1]= C:\`1`

Workarounds

The workaroung is to put the backslash in a parameter:

In[2]:=StringForm["C:`2``1`", "root", "\\"]
Out[2]= C:\root

In other contexts, it is enough to avoid escape the backquote character.

In[1]:= "\\`"

Priority

Low

Additional information

This is how WMA and Mathics process similar cases:

WMA

In[1]:=  cases = {"\\", "\ ", "\\ ", "\\\ ",                                    
  "`", "\` ", "\\` ", "\\\` "}                                                  

Out[1]= {\,  , \ , \ , `, \` , \` , \\` }

In[2]:= Table[{case, case // InputForm, StringLength[case],                     
   StringTake[case, {1}] }, {case, cases}] // TableForm                         

Out[2]//TableForm= \      "\\"      1   \

                          " "       1    

                   \      "\\ "     2   \

                   \      "\\ "     2   \

                   `      "`"       1   `

                   \`     "\` "     2   \`

                   \`     "\\` "    3   \

                   \\`    "\\\` "   3   \

Mathics

In[1]:=  cases = {"\\", "\ ", "\\ ", "\\\ ",
          "`", "\` ", "\\` ", "\\\` "}
Out[1]= {\,  , \ , \ , `, \` , \` , \\` }

In[2]:= Table[{case, case // InputForm, StringLength[case], 
           StringTake[case, {1}] }, {case, cases}] // TableForm
Out[2]//TableForm= \      "\\"       1   \
                   
                          " "        1    
                   
                   \      "\\ "      2   \
                   
                   \      "\\ "      2   \
                   
                   `      "`"        1   `
                   
                   \`     "\\` "     3   \
                   
                   \`     "\\` "     3   \
                   
                   \\`    "\\\\` "   4   \

[rocky]

The problem needs to be isolated further into whether the problem lies in the scanner, parser, or boxing.

[rocky]

For the first place we disagree, cases[[-3]], CodeTokenize gives:

In[5]:= CodeTokenize["\` "]                                                      

Out[5]= {LeafNode[Token`LinearSyntax`BackTick, \`, 
>     <|Source -> {{1, 1}, {1, 2}}|>],  
>    LeafNode[Whitespace,  , <|Source -> {{1, 2}, {1, 3}}|>]}

{LeafNode[Token`LinearSyntax`BackTick, \`,    <|Source -> {{1, 1}, {1, 2}}|>],

So there are two tokens with a backtick and whitespace
While mathics3-tokens gives:

Token('String', '"\\` "', 5)

However, I am not sure this is accurate, as one level of quotes is likely being stripped off because CodeTokenize expects a string argument. @mmatera Any thoughts on how to properly pass an indication of a string argument that is a string. The tricky part is that we don't want string interpolation of backslashes to go on in WMA beforehand.

And indeed, for the last example, CodeTokenize fails, again, I guess because the proper level of quoting does not appear.

In[18]:= CodeTokenize["\\\`"]                                                    

Out[18]= {ErrorNode[Token`Error`UnhandledCharacter, \\`, 
 
>     <|Source -> {{1, 1}, {1, 3}}|>]}

So I think for testing, instead of providing a escaped string maybe we want to build the string using <> and named characters?

I don't know.