Subject: Re: [netmod] chars that require quoting

From: Jernej Tuljak <[email-masked]>
Date: Mon, 1 Feb 2016 15:35:25 +0100

Ladislav Lhotka je 29.1.2016 ob 15:58 napisal:
>> On 29 Jan 2016, at 15:48, Martin Bjorklund <[email-masked]> wrote:
>>
>> Ladislav Lhotka <[email-masked]> wrote:
>>>> On 29 Jan 2016, at 12:55, Martin Bjorklund <[email-masked]> wrote:
>>>>
>>>> Ladislav Lhotka <[email-masked]> wrote:
>>>>> Hi,
>>>>>
>>>>> I think the following change to sec. 6.1.3 of 6020bis is needed:
>>>>>
>>>>> OLD
>>>>>   If a string contains any space or tab characters, a semicolon (";"),
>>>>>   braces ("{" or "}"), or comment sequences ("//", "/*", or "*/"), then
>>>>>   it MUST be enclosed within double or single quotes.
>>>>>
>>>>> NEW
>>>>>   If a string contains any space, tab or newline characters, a semicolon
>>>>>   (";"),
>>>>>   braces ("{" or "}"), or comment sequences ("//", "/*", or "*/"), then
>>>>>   it MUST be enclosed within double or single quotes.
>>>> I agree.
>>>>
>>>>> And what about the CR character and single or double quotes? This
>>>>> seems to be legal and results in a description argument of length 5
>>>>> (in pyang at least):
>>>>>
>>>>> description \"hi";
>>>> I don't understand what you mean.
>>> If an argument starts as unquoted, then double quotes appearing inside
>>> it (or at the end) are accepted as literals. Backslash can appear
>>> anywhere in an unquoted argument, but it is less of a problem. So the
>>> above statement rewritten with a double-quoted argument is this:
>>>
>>> description "\\\"hi\"";
>>>
>>> I think that single and double quotes (and perhaps also backslashes)
>>> shouldn't be allowed in an unquoted argument.
>> Maybe, but this is a pretty big change, and the current rules "work",
>> and we're just about to publish a version can be sent to the IESG, so
>> I'd say that it is too late for this change.
> I know, but it opens space for nasty tricks like this:
>
> $ cat quoting.yang
> module quoting {
>    namespace "http://example.com/quoting";
>    prefix qq;
>    leaf foo {
>      type string;
>      default ​"hi";
>    }
> }
> $ pyang -f yang quoting.yang
> module quoting {
>    namespace "http://example.com/quoting";
>    prefix qq;
>
>    leaf foo {
>      type string;
>      default "​\"hi\"";
>    }
> }
>
> Weird, isn't it? It is because "ZERO WIDTH SPACE" (U+200B) immediately precedes the first double quote in the argument of "default". Consequently, the argument is parsed as unquoted, and both double quotes become part of it.

I remember asking about this for 1.0. 
https://www.ietf.org/mail-archive/web/netmod/current/msg08586.html

The text is still as unclear on this matter as back then. In our 1.0 
implementation your example is syntactically invalid (missing statement 
separator, unexpected keyword). A quote always starts a string token 
(unless in comment, single quoted string or escaped in double quoted 
string).

Jernej

>
> Lada
>
>>
>> /martin
> --
> Ladislav Lhotka, CZ.NIC Labs
> PGP Key ID: E74E8C0C
>
>
>
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 1 Feb 2016 16:15:30 +0100

> On 01 Feb 2016, at 15:35, Jernej Tuljak <[email-masked]> wrote:
> 
> I remember asking about this for 1.0. https://www.ietf.org/mail-archive/web/netmod/current/msg08586.html

Your example in that message, namely 

default ";

(with just a plain space between the keyword and argument) must by all means be invalid, otherwise it would be a total disaster.
On the other hand, this is not much better, and pyang has no objections:

default \";

> 
> The text is still as unclear on this matter as back then. In our 1.0 implementation your example is syntactically invalid (missing statement separator, unexpected keyword). A quote always starts a string token (unless in comment, single quoted string or escaped in double quoted string).

In pyang it is valid. Does your implementation permit

default \; 

Lada

---

From: Jernej Tuljak <[email-masked]>
Date: Mon, 1 Feb 2016 16:38:57 +0100

> In pyang it is valid. Does your implementation permit

Yes, it does permit it since it is just a backslash, not an escape, like 
it would be inside a double quoted string. We implemented it as if 
section 6.1.2 Tokens, would contain the following sentence:

A quote character always starts a string token (unless inside a comment, a single quoted string or escaped in a double quoted string) which must be terminated with the same quote character.

It is what Juergen and Andy were claiming back then, unless I misread it.

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 1 Feb 2016 16:43:06 +0100

> On 01 Feb 2016, at 16:38, Jernej Tuljak <[email-masked]> wrote:
> 
> Yes, it does permit it since it is just a backslash, not an escape, like it would be inside a double quoted string. We implemented it as if section 6.1.2 Tokens, would contain the following sentence:
> 

And it also makes a lot of sense to me.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 1 Feb 2016 16:45:35 +0100

On Fri, Jan 29, 2016 at 01:14:27PM +0100, Ladislav Lhotka wrote:
> 
> If an argument starts as unquoted, then double quotes appearing inside it (or at the end) are accepted as literals. Backslash can appear anywhere in an unquoted argument, but it is less of a problem. So the above statement rewritten with a double-quoted argument is this:
> 
> description "\\\"hi\"";
> 
> I think that single and double quotes (and perhaps also backslashes) shouldn't be allowed in an unquoted argument.
>

Interesting, languages seem to deal with this strings like this
somewhat differently:

| lang   | input        | output | comment                                                         |
|--------+--------------+--------+-----------------------------------------------------------------|
| bash   | echo f"o"o   | foo    |                                                                 |
| tcl    | echo f"o"o   | f"o"o  |                                                                 |
| python | print f"o"o  |        | SyntaxError: invalid syntax                                     |
| perl5  | print f"o"o; |        | Bareword found where operator expected at - line 1, near ""o"o" |
| ruby   | puts f"o"o   |        | syntax error, unexpected tIDENTIFIER, expecting $end            |

It is not clear to me whether RFC 6020 or RFC 6020bis defines a
certain behaviour here. There is text about quoted strings but not
much about unquoted strings. So I would not be surprised if
implementations behave differently here.

The text in RFC 6020bis also defines backslash behaviour only inside
quoted strings. It is left to an implementor's imagination what f\"oo
resolves to I think (and if you comine things, what f\"o"o means).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 01 Feb 2016 20:08:00 +0100 (CET)

Juergen Schoenwaelder <[email-masked]> wrote:
> resolves to I think

Do you mean b/c of the double-qoute character?

Why would an implementor think that the backslash character is special
in any way in an unquoted string?  IMO it would be very odd to think
that since the text doesn't mention that backslash is special in an
unquoted string, it probably means that an implementation can choose
to interpret it in anyway he likes.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 1 Feb 2016 20:25:09 +0100

> to interpret it in anyway he likes.

If a specification is not explicit enough, then people often implement
what they find implemented in other similar contexts. This means the
code often ends up reflecting which tools an implementor was familiar
with. In a shell, echo f\"oo gives you f"oo (and the same is true for
Tcl, which has otherwise the behavior you implemented for pyang,
treating " as a regular character in an unquoted string.)

It would help to know what implementations do with identifiers like
f"o"o and f\"o\"o. If they do different things, then there is likely
some evidence that implementors arrived at different conclusions.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 01 Feb 2016 20:48:28 +0100 (CET)

> some evidence that implementors arrived at different conclusions.

I don't mind a clarification.  How about:

OLD:

  If a string contains any space, tab, or newline characters, a
  semicolon (";"), braces ("{" or "}"), or comment sequences ("//",
  "/*", or "*/"), then it MUST be enclosed within double or single
  quotes.

NEW: 

  If a string contains any space, tab, or newline characters, a
  semicolon (";"), braces ("{" or "}"), or comment sequences ("//",
  "/*", or "*/"), then it MUST be enclosed within double or single
  quotes.  If a string does not contain any of these characters, it
  MAY be unquoted.

  Within an unquoted string, every character is preserved.  Note that
  this means that the backslash, single quote, and double quote
  characters can occur in an unquoted string.

/martin

---

From: Jernej Tuljak <[email-masked]>
Date: Tue, 2 Feb 2016 09:51:44 +0100

>    characters can occur in an unquoted string.

I don't think this can be implemented, like I did not in the linked 
archive. It does not make clear whether these are two statements or a 
single one:

    default ";
    units ";

What about this:

    default ";//foo";
    units \";

This is not the C and C++ way of doing things (possibly not even SMIng 
way). These are the only languages mentioned in the document (Section 
6). It also does not promote readability and makes lexers unnecessarily 
complex.

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 2 Feb 2016 10:04:24 +0100

> On 01 Feb 2016, at 20:48, Martin Bjorklund <[email-masked]> wrote:
> 

For the reasons that I explained earlier, I think that allowing single and double quotes in unquoted strings is a bug. This text makes it into a feature, so I am objecting to it.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 2 Feb 2016 10:16:51 +0100

> On 02 Feb 2016, at 09:51, Jernej Tuljak <[email-masked]> wrote:
> 
> I don't think this can be implemented, like I did not in the linked archive. It does not make clear whether these are two statements or a single one:
> 
>   default ";
>   units ";
> 
> 
> What about this:
> 
>   default ";//foo";
>   units \";
> 
> 
> This is not the C and C++ way of doing things (possibly not even SMIng way). These are the only languages mentioned in the document (Section 6). It also does not promote readability and makes lexers unnecessarily complex.

+1

I think the safest solution would be to disallow in unquoted strings all characters that have a special meaning anywhere in YANG syntax. This can hardly cause any troubles to module authors.

BTW, if we wanted to be extremely liberal, I don't understand why the end-comment sequence "*/" is not allowed.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 02 Feb 2016 12:16:27 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> I think the safest solution would be to disallow in unquoted strings
> all characters that have a special meaning anywhere in YANG
> syntax. This can hardly cause any troubles to module authors.

Fine with me.  Something like:

NEW: 

  If a string contains any space, tab, or newline characters, a single
  or double quote, semicolon (";"), braces ("{" or "}"), or comment
  sequences ("//", "/*", or "*/"), then it MUST be enclosed within
  double or single quotes.  If a string does not contain any of these
  characters, it MAY be unquoted.

  Within an unquoted string, every character is preserved.  Note that
  this means that the backslash character does not have any special
  meaning in an unquoted string.

It should be noted that this may be viewed as a backwards incompatible
change, but I think it is similar to the change for escaped characters
in double quoted string ("backwards incompatible bugfix").

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 2 Feb 2016 12:25:57 +0100

> On 02 Feb 2016, at 12:16, Martin Bjorklund <[email-masked]> wrote:
> 

I like this.

> in double quoted string ("backwards incompatible bugfix").

Yes, I think so.

Thanks, Lada

---

From: Jernej Tuljak <[email-masked]>
Date: Tue, 2 Feb 2016 13:04:45 +0100

+1

>> in double quoted string ("backwards incompatible bugfix").

This is nothing compared to the change of the accessible tree for output.

Jernej

---

From: Andy Bierman <[email-masked]>
Date: Tue, 2 Feb 2016 09:25:02 -0800

On Tue, Feb 2, 2016 at 4:04 AM, Jernej Tuljak <[email-masked]>
wrote:

>>>>> way). These are the only languages mentioned in the document (Section 6).
>>>>> It also does not promote readability and makes lexers unnecessarily complex.
>>>>>
>>>> +1
>>>>
I don't understand this text.
The parser looks for certain tokens in specific contexts.

  container A;container B;container C;

  container foo{container bar{container baz;}}

  augment /foo/bar/baz{container Z;}

pyang and yangdump-pro handle these unquoted strings correctly.

I think your text is supposed to mean that chars with special meanings
in specific contexts need to be quoted to use them as plain chars without
any special meaning.

I strongly object to this new text since it tells the YANG author they
cannot use unquoted strings like the examples above.

Andy

>>> BTW, if we wanted to be extremely liberal, I don't understand why the
>>>> end-comment sequence "*/" is not allowed.
>>>>
>>>> Lada
>>>>
>>>> Jernej
>>>>>
>>>>> /martin
>>>>>>

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 2 Feb 2016 18:50:32 +0100

> On 02 Feb 2016, at 18:25, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> On Tue, Feb 2, 2016 at 4:04 AM, Jernej Tuljak <[email-masked]> wrote:

Why do you think there is anything in the new text to this effect? All of your examples remain perfectly OK. The only change compared to 6020bis-09 is that the newline, single- and double-quote characters cannot appear in unquoted argument strings.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Tue, 2 Feb 2016 20:02:58 -0800

On Tue, Feb 2, 2016 at 9:50 AM, Ladislav Lhotka <[email-masked]> wrote:

> Why do you think there is anything in the new text to this effect? All of
> your examples remain perfectly OK. The only change compared to 6020bis-09
> is that the newline, single- and double-quote characters cannot appear in
> unquoted argument strings.
>
>

*If a string contains* any space, tab, or newline characters, a single
>   or double quote, *semicolon (";"),* braces ("{" or "}"), or comment
>   sequences ("//", "/*", or "*/"), t
*hen it MUST be enclosed within>   double or single quotes. *

This is not true;  The example above contains valid unquoted strings that
contain a semicolon.

The only strings that MUST be encoded in quotes are a single quote or
double quote.
All parsers will look for the closing quote and there won't be one.

IMO the only next needed:

NEW:

   If a string contains any single quote characters, then it MUST be
enclosed within
   double quotes.  If a string contains any double quote characters, then
it MUST be
   enclosed within single quotes, or encoded within double quotes using an
escape
   character '\', followed by the double quote character '"'.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 3 Feb 2016 06:59:17 +0100

> On 03 Feb 2016, at 05:02, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> >   double or single quotes. 
> This is not true;  The example above contains valid unquoted strings that contain a semicolon.

Huh? The text is about characters in unquoted *argument* strings. The semicolons in your examples aren't syntactically part of an argument. This hasn't changed: semicolons and curly braces were not permitted in an unquoted argument already in YANG 1.0.

Lada

> 
> The only strings that MUST be encoded in quotes are a single quote or double quote.
>    If a string contains any single quote characters, then it MUST be enclosed within
>    double quotes.  If a string contains any double quote characters, then it MUST be
>    enclosed within single quotes, or encoded within double quotes using an escape

---

From: Jernej Tuljak <[email-masked]>
Date: Wed, 3 Feb 2016 09:03:30 +0100

I agree.

   container A ; container B ; container C ;

   container foo { container bar { container baz ; } }

   augment /foo/bar/baz { container Z ; }

The semantics of the above should be equivalent to Andy's example. This 
is probably some sort of a misunderstanding.

The new text is only adding quotes and newlines to the list of 
characters that need to be inside quoted strings in order to not be 
recognized as a token, a token start or a token separator.

Jernej