Subject: Re: [netmod] Tokenizing and strings

From: "Jernej Tuljak" <[email-masked]>
Date: Fri, 10 Jun 2016 10:45:58 +0200

> -----Original Message-----
> From: netmod [mailto:[email-masked]] On Behalf Of Dale R.
> Worley
> Sent: Friday, June 10, 2016 12:04 AM
> To: Martin Bjorklund <[email-masked]>
> Cc: [email-masked]
> Subject: [netmod] Tokenizing and strings (was: Gen-ART IETF Last Call 
> review
> of draft-ietf-netmod-rfc6020bis-12 (part 2))
>
> At this point, I think there are some aspects of the use of strings and
> identifiers that aren't completely specified, but I'd have to check the 
> current
> text to know for certain.  However, that's a minor matter.
>
> The issue that concerns me is that the ABNF doesn't specify what is 
> allowed
> as a string.  I'm used to programming language definitions, where the
> grammar is specified quite rigidly, to the point that the ABNF can be 
> input to
> a parser generator.  In this document, the ABNF is quite complete except 
> for
> a specification of strings.  On the other hand, the text description of 
> strings
> seems to be sufficient for an implementer, so we don't actually need to
> provide ABNF.  My strong preference is to provide a complete ABNF, as is 
> the
> norm for programming languages.
>
> The following is a complete ABNF for Yang strings.  Of course, it's a bit
> complicated, because the definition of strings in Yang actually is a bit
> complicated.
>
>    string              = unquoted-string / quoted-string
>
>    unquoted-string     = *unquoted-item ( unquoted-item /
>                                           "/" /
>                                           "*" *"*" )
>                          ;; a sequence of one or more characters from
>                          ;; ordinary-char / "/" / "*", not containing
>                          ;; "//", "/*", or "*/"

This rule allows the "*/" sequence to appear inside unquoted strings!

Less is more in case of YANG strings and grammars in general. You can be as descriptive as you want in text, but not in pure ABNF.
 
You are also forgetting the fact that the existing 'string' production expects  an "unquoted string as returned by the scanner" (no quotes, no concats, no pretty printing whitespace).

Jernej

>
>    unquoted-item       = ordinary-char /
>                          "/" ordinary-char /
>                          "*" *"*" ordinary-char
>
>    ordinary-char       = < any character matching yang-char, except >
>                          < space, tab, newline, carriage return,    >
>                          < semicolon, left brace, right brace,      >
>                          < slash, and asterisk                      >
>
> *** Hmmm, is an unaccompanied CR allowed in an unquoted string?  If so,
> "ordinary-char" and my previous text regarding line breaks need to be
> amended.
>
>    quoted-string      = ( single-quoted-string / double-quoted-string )
>                         *( optsep "+" optsep
>                            ( single-quoted-string / double-quoted-string )
> )
>
> (I think you said that there can be whitespace around + but not comments.)
>
>    single-quoted-string = SQUOTE *sq-char SQUOTE
>
>    sq-char            = < any character matching yang-char, except >
>    		        < SQUOTE                                   >
>
>    double-quoted-string = DQUOTE *dq-item DQUOTE
>
>    dq-item            = dq-char /
>    		      	"\n" /
> 			"\t" /
> 			"\" DQUOTE /
> 			"\\"
>
>    dq-char            = < any character matching yang-char, except >
>    		        < DQUOTE and backslash                     >
>
> (The existing production for yang-string is removed.)
>
>    ;; any Unicode or ISO/IEC 10646 character including tab, carriage
>    ;; return, and line feed, but excluding the other C0 control
>    ;; characters, the surrogate blocks, and the noncharacters.
>    yang-char = %x09 / %x0A / %x0D / %x20-D7FF /
>    [continuing as before]
>
> Dale
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: "Jernej Tuljak" <[email-masked]>
Date: Fri, 10 Jun 2016 11:06:40 +0200

> From: netmod [mailto:[email-masked]] On Behalf Of Jernej Tuljak
> Sent: Friday, June 10, 2016 10:46 AM
> To: 'Dale R. Worley' <[email-masked]>; 'Martin Bjorklund' <mbj@tail-
> f.com>
> Cc: [email-masked]
> Subject: Re: [netmod] Tokenizing and strings
> 
> > review of draft-ietf-netmod-rfc6020bis-12 (part 2))
> >
> > At this point, I think there are some aspects of the use of strings
> > and identifiers that aren't completely specified, but I'd have to
> > check the current text to know for certain.  However, that's a minor
> > matter.
> >
> > allowed as a string.  I'm used to programming language definitions,
> > where the grammar is specified quite rigidly, to the point that the
> > ABNF can be input to a parser generator.  In this document, the ABNF
> > is quite complete except
> 
> > for
> > a specification of strings.  On the other hand, the text description
> > of strings seems to be sufficient for an implementer, so we don't
> > actually need to provide ABNF.  My strong preference is to provide a
> > complete ABNF, as is
> 
> > the
> > complicated, because the definition of strings in Yang actually is a
> > bit complicated.
> >

Scratch that. I had a bug in my test ANTLR4 grammar.

Jernej

> 
> Less is more in case of YANG strings and grammars in general. You can be as
> descriptive as you want in text, but not in pure ABNF.
> 
> You are also forgetting the fact that the existing 'string' production expects
> an "unquoted string as returned by the scanner" (no quotes, no concats, no
> pretty printing whitespace).
> 
> Jernej
> 
> 
> >
> > *** Hmmm, is an unaccompanied CR allowed in an unquoted string?  If
> > so, "ordinary-char" and my previous text regarding line breaks need to
> > be amended.
> >
> >                            ( single-quoted-string /
> > double-quoted-string

---

From: [email-masked] (Dale R. Worley)
Date: Sun, 12 Jun 2016 20:52:41 -0400

"Jernej Tuljak" <[email-masked]> writes:
> You are also forgetting the fact that the existing 'string' production
> expects  an "unquoted string as returned by the scanner" (no quotes,
> no concats, no pretty printing whitespace).

I must admit I've never understood what that bit of the text means.

Dale