Subject: Re: [netmod] [Gen-art] Gen-ART IETF Last Call review of draft-ietf-netmod-rfc6020bis-12 (part 1)

From: [email-masked] (Dale R. Worley)
Date: Mon, 06 Jun 2016 17:34:21 -0400

(This is the first part of my response, which I am sending seperately to
speed things up.)

Martin Bjorklund <[email-masked]> wrote:
> [email-masked] (Dale R. Worley) wrote:
> > Martin Bjorklund <[email-masked]> wrote on Mon, 23 May 2016 15:43:09 +0200 (CEST):
> > > [email-masked] (Dale R. Worley) wrote:
> > 
> > Almost all of these points I'm happy with the authors fixing in the
> > manner they suggest.  The points where I have further comment are:
> > 
> > > > The incompatibilities should be marked whether the old, incompatible
> > > > usage always causes an error "at compile time", "at run time", or
> > > > changed behavior.
> > > 
> > > Ok, but I'd like to avoid the term "compile-time" here.
> > 
> > True...  The distinction I'm looking for is "Incompatible usage will
> > cause an error report before the software is put into operation."
> > vs. "Incompatible usage will cause an error report when the software is
> > in operation." vs. "Incompatible usage will cause a difference in
> > behavior without an error report."
> 
> The spec defines what is legal YANG.  How this is enforced by tools
> must be up to the tools, right?

It depends.  When reading about incompatibilities, it's really nice to
be informed what will happen.  E.g., if the compiler will catch it,
you just recompile the code and watch for error messages.  If it will
cause incompatible run-time behavior, you have to inspect the code,
which is a lot of work.

> > > How about:
> > > 
> > >    The following changes are not backwards compatible with YANG version
> > >    1:
> > > 
> > >    o  Changed the rules for the interpretation of escaped characters in
> > >       double quoted strings.  This is an backwards incompatible change
> > >       from YANG version 1.  A module that uses a character sequence that
> > >       is now illegal must change the string to match the new rules.  See
> > >       Section 6.1.3 for details.
> > > 
> > >    o  An unquoted string cannot contain any single or double quote
> > >       characters.  This is an backwards incompatible change from YANG
> > >       version 1.  A module that uses such quote characters must change
> > >       the string to match the new rules.  See Section 6.1.3 for details.
> > 
> > I assume that the first two situations will cause an error report before
> > the module is put into production.
> > 
> > >    o  Made noncharacters illegal in the built-in type "string".  This
> > >       change affects the run-time behavior of YANG-based protocols.
> > 
> > What happens if you attempt to use a noncharacter in a string?
> 
> This depends on the protocol.  noncharacters are illegal.  Maybe in
> protocol A you can't even encode noncharacters, but in protocol B
> you'll get a protocol error from the other side if you send
> noncharacters.

Really, the question is "What does 'illegal' mean?"  Of course, it
means that you shouldn't do it, but does it also contain assertions
that some particular stage of processing will explicitly flag it?

> > It might be worth inserting a definition of "node" as shorthand for
> > "data node", as there seems to be no definition of "node" alone, but it
> > is commonly used to mean "data node".  In particular, the reader needs
> > to be fully aware that "node" is a node in the schema tree, not an
> > instantiation thereof.
> 
> But saying that "node" means "data node" is not correct.  Sometimes
> the text talk about "schema nodes" etc.  I think it is better to
> qualify the usage of "node" where necessary to avoid the ambiguity.

OK, I see.  But since I was getting confused, it suggests that there
are various places where "node" is used where it wasn't clear from
context whether it was a data node or a schema node.

> > > > It would be useful to insert a note somewhere that all data is either
> > > > "configuration" or "state" data.  It's hard to learn that now, because
> > > > the terms "configuration data" and "state data" are defined only by
> > > > reference to RFC 6241.
> > > 
> > > But this is not strictly correct.  For example RPC input is neither
> > > config nor state.  Section 3 has:
> > > 
> > >   o data tree: An instantiated tree of any data modeled with YANG,
> > >     e.g., configuration data, state data, combined configuration and
> > >     state data, RPC or action input, RPC or action output, or
> > >     notification.
> > 
> > Section 4.1 contains a similar statement.
> > 
> > I think the problem I'm having is with the first sentences of 4.2.3:
> > 
> >    YANG can model state data, as well as configuration data, based on
> >    the "config" statement.
> > 
> > At first read, this suggests that data is partitioned into
> > "configuration data" and "state data".  What's really happening is that
> > there are various sorts of data, but some contexts admit a mixture of
> > configuration and state data, so the data model definitions of those
> > contexts requires that individual data items can be flagged as
> > configuration vs. data.  Maybe starting 4.2.3 like this would work:
> > 
> >    In some contexts, data modeled by YANG can contain mixtures of state
> >    data and configuration data, based on the "config" statement.  When a
> >    node is tagged with "config false", its subhierarchy is flagged as
> >    state data.
> 
> I think that this might be a bit confusing (which contexts?  what
> exacyly is this mixture?) and too detailed for this introductory
> section.   I would perfer to keep the original text.

That leaves the problem that there are forms that are neither state
data nor configuration data.

Also, the phrasing is odd in that it seems to assume that
configuration data is sort of a default condition and state data is an
exception.  I think what was intended is

    YANG can differentiate state data and configuration data within
    one instantiated tree, based on the "config" statement.

> > > > - section 5.1.2
> > > > 
> > > >    YANG allows modeling of data in multiple hierarchies, where data may
> > > >    have more than one top-level node.  Models that have multiple top-
> > > >    level nodes are sometimes convenient, and are supported by YANG.
> > > > 
> > > > Any single module seems to define only one data structure, the one
> > > > whose elements are the items listed in the module, and Yang doesn't
> > > > seem to provide for any sort of "model" definition other than a
> > > > module.  Or are the nodes at the top level within the module all
> > > > usable as independent data structures?
> > > 
> > > The term "data structure" isn't used in the document.  YANG allows
> > > a module to define more than one subtree.
> > 
> > OK, I think I see where I was confused.  I think that would have been
> > avoided if the 5.1.2 started:
> > 
> >    YANG allows modeling of data in multiple hierarchies.  Each top-level
> >    data node in a module defines an independent hierarchy.  Models that
> >    have ...
> 
> What does "independent" imply?  They are not necessarily semantically
> independent.  Maybe s/independent/separate/ ? 

Yes, "separate" would work well.

Here, I'll mention again this point from another e-mail:  The second
sentence of section 4.1 suggests that a Yang module defines *one*
hierarchy of data, whereas it seems like any top-level data node in a
module has meaning separately and can be instantiated seperately.

> > > > - section 5.5
> > > > 
> > > > This section talks a great deal about why scoping is done, but isn't
> > > > specifically clear what the rules are.  (Traditionally, the rule is
> > > > stated as either what range of source text a particular definition is
> > > > visible in, or given an identifier use, how the matching definition is
> > > > found.)
> > > > 
> > > >    Scoped definitions MUST NOT shadow definitions at a higher scope.  A
> > > >    type or grouping cannot be defined if a higher level in the schema
> > > >    hierarchy has a definition with a matching identifier.
> > > > 
> > > > I think the second sentence is confusing, because (to me) the term
> > > > "schema hierarchy" implies the data tree structure, whereas the
> > > > scoping rule seems to be intended to be entirely lexical.  In either
> > > > case, this needs to be clarified.
> > > 
> > > The first sentence says:
> > > 
> > >   Typedefs and groupings may appear nested under many YANG statements,
> > >   allowing these to be lexically scoped by the hierarchy under which
> > >   they appear.
> > > 
> > > Maybe it would help to move paragraph 2 and 3 to the end, so that the
> > > motivational text is at the end of the section?
> > 
> > It looks like current paragraphs 1, 4, and 5 are prescriptive, and 2 and
> > 3 are motivational.  Putting 1, 4, and 5 together should make them
> > clearer.  But since paragraph 2 starts with "Scoping...", it really
> > needs to be after 1.  So, yes, moving 2 and 3 to the end seems like the
> > clearest order.  Does it make sense to you to put all of the motivation
> > at the end of this section?
> 
> I prefer to have motivational text before the rules ("why" before
> "how").  Hmm.  The preferred order is "what", "why", "how", and that
> is really the order we currently have (1 is "what", 2 + 3 is "why" and
> 4 +5 is "how").
> 
> But obviously this was confusing to you, so we probably need to make
> some clarification.
> 
> Since the problem was with the term "schema hierarchy", maybe we could
> change this to "statement hierarchy" instead, to stress the fact that
> it is lexical scoping?
> 
> [sorry for missing your original point]

Yes, I think that will do it.  There are two instances of "hierarchy"
in para. 1, and at least the first one would be clearer if it was
"statement hierarchy".  And in para. 4, it seems like "schema
hierarchy" is actually incorrect, and should be "statement hierarchy".

> > > > - section 5.6.5
> > > > 
> > > > I suspect that if a server implements module A, and A imports B, the
> > > > server is not required to implement B.  (Even if A references
> > > > groupings in B.)  The answer to that should probably be stated
> > > > explicitly.
> > > 
> > > It is not so simple.  The rest of the text defines the rules for when
> > > the server is required to implement B.
> > 
> > Yes, I was wrong there.  (I can't see why I made that mistake.)
> > 
> > > > Is this the correct way to specify multiple revisions of this module?
> > > > Or is this an informal notation for the two module definitions:
> > > > 
> > > >      module b {
> > > >        yang-version 1.1;
> > > >        namespace "urn:example:b";
> > > >        prefix "b";
> > > > 
> > > >        revision 2015-04-04;
> > > >        revision 2015-01-01;
> > > > 
> > > >        typedef myenum {
> > > >          type enumeration {
> > > >            enum zero; // added in 2015-01-01
> > > >            enum one;  // added in 2015-04-04
> > > >          }
> > > >        }
> > > > 
> > > >        container x {  // added in 2015-01-01
> > > >          container y; // added in 2015-04-04
> > > >        }
> > > >      }
> > > > 
> > > >      module b {
> > > >        yang-version 1.1;
> > > >        namespace "urn:example:b";
> > > >        prefix "b";
> > > > 
> > > >        revision 2015-01-01;
> > > > 
> > > >        typedef myenum {
> > > >          type enumeration {
> > > >            enum zero; // added in 2015-01-01
> > > >          }
> > > >        }
> > > > 
> > > >        container x {  // added in 2015-01-01
> > > >        }
> > > >      }
> > > > 
> > > > If it is the latter, it would help the new reader to explain the
> > > > convention (as no example of a multi-revision module is given in the
> > > > document).
> > > 
> > > The comments are not required.  There is no correct (mandated) way to
> > > specify which changes are done in a new revision.
> > 
> > What I was concerned with is what does the above module statement do?
> > If I understand correctly, it defines the 2015-04-04 revision of b, and
> > also tells us that a 2015-01-01 revision exists, but does not define
> > that revision.  Implicitly, there must be a .yang file somewhere
> > containing the module statement that defines the 2015-01-01 revision of
> > b.
> > 
> > But despite that the text talks about the 2015-01-01 revision ('can
> > implement revision "2015-01-01" or "2015-04-04" of module "b"'), that
> > file is not included in the example, which is not what I expected, which
> > led me to wonder if the 2015-01-01 revision was somehow also defined by
> > the presented "module b" statement.  I would have had less trouble with
> > the example if the definition of the earlier revision of b was also
> > given.
> 
> Ok, I see what you mean.  This makes sense.  I'll add 
> 
>   module b {
>     yang-version 1.1;
>     namespace "urn:example:b";
>     prefix "b";
> 
>     revision 2015-01-01;
> 
>     typedef myenum {
>       type enumeration {
>         enum zero;
>       }
>     }
> 
>     container x {
>     }
>   }
> 
> and
> 
>   module c {
>     yang-version 1.1;
>     namespace "urn:example:c";
>     prefix "c";
> 
>     revision 2015-02-02;
> 
>     typedef bar {
>       ...
>     }
>   }
> 
> 
> to the example.

Yes, I think that will make it all clear.

> > > > It is worth noting (either here or in 6.1.3) how line-breaks inside
> > > > quoted strings are transcribed into the string's value.  As now
> > > > written, it seems that the line-break is transcribed identically to
> > > > how it is represented in the source.  That means (1) If the source is
> > > > recoded with the other type of line break, the semantics of the Yang
> > > > code change; and (2) if the source uses line-breaks of one type (CRLF
> > > > or LF), only that type can be directly transcribed into string values.
> > > > (But regardless of the source line-breaks, an LF can be transcribed
> > > > into a double-quoted string with "\n".  But a CRLF cannot be
> > > > transcribed into a double-quoted string with escape sequences.  Was
> > > > that intended, or was "\r" intended to be legal?)
> > 
> > Don't forget this.

I was wondering that (as the text is written) the value of a quoted
string that includes a line end will change if the line ends are
changed (say, by moving the Yang file to a different host).  Normally
computer languages are defined so that changing how the program is
"represented", e.g., by transforming it into a form with different
line-ends, won't change the semantics of the program.

But if it's true, I seems desirable to provide an explicit warning
about it.

> Adding "\r" would be backwards incompatible with YANG 1.

I don't see why that is.  As far as I can tell from RFC 6020, "\r" is
not a defined escape in Yang 1, so existing Yang 1 code can't use it.
True, the new draft explicitly states "It is an error if any other
character follows the backslash character.", but RFC 6020 doesn't seem
to allow "\r" to mean anything.  Or is it that in practice, Yang 1
allows people to write "\r", which is interpreted as backslash-r?

> You can always get the same effect by using single quoted strings.

I don't see how you can get a single CR using single-quoted strings,
because line ends are only allowed to be LF and CR-LF; writing a CR
without an LF following it isn't allowed.  Or is it that in practice,
Yang 1 allows people to use CR alone, and it is not interpreted as a
line-end?

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Mon, 06 Jun 2016 18:28:20 -0400

(This is the second part of my response.)

> > > > - section 6.1
> > > > 
> > > >    This section details the rules for recognizing tokens from an input
> > > >    stream.
> > > > 
> > > > Generally, language definitions intersperse the narrative text with
> > > > the relevant grammar definitions.  Yang's statement grammar is simple
> > > > enough that one doesn't need to see the context-free part of the
> > > > grammar to understand the narrative for statements.  But when reading
> > > > about tokenization, not having the grammar presented at the same time
> > > > is quite a burden.  I'd recommend duplicating the relevant productions
> > > > from section 14 into the subsections of section 6.
> > > > 
> > > > There is some sort of exposition problem.  The result of
> > > > "tokenization" is that the sequence of characters of the source is
> > > > converted into a sequence of "tokens".  Then some subset of the tokens
> > > > is discarded as being non-significant (e.g., whitespace and comments),
> > > > and the remainder is parsed with a context-free grammar.  Here I can't
> > > > figure out what the set of tokens is.  Looking at the grammar in
> > > > section 14, it seems to be a context-free grammar on characters.  But
> > > > that implies that there is no separate tokenization phase.
> > > > 
> > > > An example that shows the problems:
> > > > 
> > > >    mod:ext
> > > > 
> > > > Is this one token, which is also an extension keyword, or is it a
> > > > sequence of three tokens?
> > > 
> > > The text says:
> > > 
> > >   A token in YANG is either a keyword, a string, a semicolon (";"), or
> > >   braces ("{" or "}").
> > > 
> > > and:
> > > 
> > >   A keyword is [...] or a prefix identifier, followed by a colon
> > >   (":"), followed by a language extension keyword.
> > > 
> > > So "mod:ext" is one token.
> > 
> > Certainly it can be one token.  My question is how do verify that it is
> > not a string?  I think that may be the origin of my confusion here is
> > that I haven't spotted a clear syntax for unquoted string.  In most
> > programming languages, mod:ext would be parsed as an identifier, a
> > colon, and an identifier.  In YANG, identifiers are usually tokenized as
> > strings, so I ask whether YANG tokenizes it as a string, a colon, and a
> > string.
> > 
> > Looking at the beginning of 6.1.3, it doesn't appear that an unquoted
> > string is forbidden from containing a colon.
> > 
> > I think that the underlying problem is that I'm not clear on what gets
> > tokenized as an unquoted string.
> 
> Note that this is legal YANG:
> 
>    leaf type {
>      type string;
>    }

So keywords aren't reserved; they can also be used as identifiers.

> I think there are two ways to look at this.  Either we describe the
> tokenizer as being context-dependent, or we describe the "argument" in
> a "statement" to be a "string or keyword".
> 
> In the latter case maybe we can do:
> 
> OLD:
> 
>   If a string contains any space, tab, or newline characters, a single
>   or double quote character, a semicolon (";"), braces ("{" or "}"),
>   or comment sequences ("//", "/*", or "*/"), then it MUST be enclosed
>   within double or single quotes.
> 
> NEW:
> 
>   An unquoted string is any sequence of characters that does not start
>   with a double or single quote character, is not a keyword, and does
>   not contain any space, tab, or newline characters, a single or
>   double quote character, a semicolon (";"), braces ("{" or "}"), or
>   comment sequences ("//", "/*", or "*/").

That's a lot clearer.  Though you can shorten it to:

   An unquoted string is any sequence of characters that is not a
   keyword, and does not contain any space, tab, or newline
   characters, a single or double quote character, a semicolon (";"),
   braces ("{" or "}"), or comment sequences ("//", "/*", or "*/").

> In section 6.3 we must also do:
> 
> OLD:
> 
>    The argument is a string, as defined in Section 6.1.2.
> 
> NEW:
> 
>    The argument is a string or a keyword, as defined in Section 6.1.2.

If I understand correctly, the tokens of Yang (as the term is usually
used in programming languages) are:

    whitespace (which is ignored)
    comments (which is ignored)
    single-quoted strings
    double-quoted strings
    unquoted strings (including keywords)
    ;
    {
    }

>From the point of view of the tokenizer, these tokens fall into the
obvious classes:

	type	unquoted string
	"type"	double-quoted string
	abc	unquoted-string
	"abc"	double-quoted string
	'---'	single-quoted string

I'm not quite sure how they are classified from the parser's point of
view, though.

				type	"type"	abc	"abc"	'---'

Is a string?			?	Y	?	Y	Y
(Can it appear as the
argument of "description"?)

Is a keyword?			Y	?	N	N	N
(Can it appear as the first
token of some statement?)

Is an identifier?		Y	?	Y	?	N
(Can it appear as the second
token of a type statement?)

Usually programming languages use the particular syntax of different
types of tokens to determine where they can be used in the
context-free grammar.  Yang seems to be more relaxed, but I'm not sure
whether it is so relaxed thay any of the types of string tokens can be
used anywhere.

> > > > -- it must be an unquoted string.
> > > > 
> > > >    If a double-quoted string contains a line break followed by space or
> > > >    tab characters that are used to indent the text according to the
> > > >    layout in the YANG file, this leading whitespace is stripped from the
> > > >    string, up to and including the column of the double quote character,
> > > >    or to the first non-whitespace character, whichever occurs first.  In
> > > >    this process, a tab character is treated as 8 space characters.
> > > > 
> > > > This description isn't quite careful enough.  Better:
> > > > 
> > > >    If a double-quoted string contains a line break followed by space or
> > > >    tab characters, an initial part of this whitespace is removed from the
> > > >    string.  The amount removed is the longest prefix whose width is no
> > > >    larger than the width of the prefix of Yang source line containing
> > > >    the opening double quote character of the string to and including the
> > > >    opening double quote character.  For this purpose, the width of a
> > > >    tab character is 8 and the width of any other character is 1.
> > > > 
> > > > This does assume that all tabs are considered to have width 8, that
> > > > is, tabs do not have the usual semantics of "advance to the next
> > > > column that is divisible by 8".  That will sometimes cause unexpected
> > > > results, e.g., if some source lines start with SPC TAB.  (Consider
> > > > that whitespace before a line break is removed, which suggests the
> > > > intention is that the value of the string should depend only on its
> > > > visual appearance.)
> > > > 
> > > > Also, we're using the convention that "whitespace" does NOT include CR
> > > > or LF, which is not always how the term is used.  Perhaps a definition
> > > > of "whitespace" should be put in section 3.
> > > > 
> > > > There is also the special case:
> > > > 
> > > >    SPC " LF
> > > >    TAB x "
> > > > 
> > > > Is the initial TAB of the second line to be removed or not?  There is
> > > > no whitespace removal in the second line that will exactly reach the
> > > > opening double quote.  As I've written it, the TAB is not removed.
> > 
> > Don't forget this ugly special case.
> 
> So, let's follow the rules.  We need to trim to the column of the
> double quote character (2).  The second line starts with "space or
> tab" so we do whitespace trimming, while treating the tab as 8
> spaces.  So from 8 spaces we subtract 2, and get the resulting string
> of 6 characters:
> 
>   LF SPC SPC SPC SPC SPC SPC x

OK, but that process wasn't clear to me.  I take it that any tab that
appears before the starting double-quote counts as 8 spaces, and any
tab that needs to be examined for deletion is turned into 8 spaces --
but any other tabs in the string are unconverted.

I think it would be clearer to insert "starting" where I've indicated
it, and replace the final sentence:

   If a double-quoted string contains a line break followed by space or
   tab characters that are used to indent the text according to the
   layout in the YANG file, this leading whitespace is stripped from the
   string, up to and including the column of the >starting< double quote character,
   or to the first non-whitespace character, whichever occurs first.
   In this process, any tab character before the starting double quote
   character is treated as 8 spaces.  Any tab character in a succeeding
   line that must be examined to for stripping is first converted into 8
   spaces.

> > Actually, there is a somewhat subtle problem:  If I say "the system can
> > sort them any way it wants", I am asserting that *there is a sorting
> > order*.  Which means that if value A is put before value B at one time,
> > then if values A and B are in the list at some other time, A will
> > precede B.
> 
> The next sentences says:
> 
>   An implementation SHOULD use the same order for the same data,
>   regardless of how the data were created.  Using a deterministic
>   order will make comparisons possible using simple tools like "diff".

OK, I'm willing to go with that.  I mis-read the application of those
sentences through an even more arcane ambiguity in the term "the same
data".  But I'm willing to ignore that.

> > > > - section 7.21.4
> > > > 
> > > >    The "reference" statement takes as an argument a string ...
> > > > 
> > > > Perhaps s/a string/a human-readable string/.
> > > 
> > > "string" refers to the YANG token "string".  The same wording is used
> > > across the document for all arguments.
> > 
> > I was thinking that it is a string, but in this particular case, it is
> > supposed to be human-readable, whereas strings in other contexts aren't
> > expected to be.
> 
> Ok.  Maybe:
> 
> OLD:
> 
>   The "reference" statement takes as an argument a string that is used
>   to specify a textual cross-reference to an external document,
> 
> NEW:
> 
>   The "reference" statement takes as an argument a string that is used
>   to specify a human-readable cross-reference to an external document,

Or even "is a human-readable cross-reference ...", but either is OK
with me.

> > > > - section 7.21.5
> > > > 
> > > > Note that if a data definition has both an "if-feature" and a "when",
> > > > then the "if-feature" is tested first.
> > > > 
> > > >    If the XPath expression references any node that also has associated
> > > >    "when" statements, these "when" expressions MUST be evaluated first.
> > > >    There MUST NOT be any circular dependencies in these "when"
> > > >    expressions.
> > > > 
> > > > I think this could be better phrased:
> > > > 
> > > >    If the XPath expression references any node that also has
> > > >    associated "when" statements, then the "when" expressions of the
> > > >    referenced nodes MUST be evaluated first.  There MUST NOT be any
> > > >    circular dependencies among "when" expressions.
> > > 
> > > Ok to the last sentence.  Do you think that the word "these" in the
> > > first sentence is ambigious?
> > 
> > I must have thought it was unclear when I read it, otherwise I would not
> > have suggested changing it.  But reading it again, I think that there is
> > no ambiguity.  Perhaps it would be a little clearer to use 'those "when"
> > expressions' rather than 'these "when" expressions'.  (I can't explain
> > clearly why "those" seems less ambiguous than "these".)
> 
> Ok, as a non-native english speaker I trust you that "those" is better.

I can't tell that you're non-native.  Perhaps leave it as is and let
the RFC Editor review it.

> > By implication, the leafref's value is considered to be a pointer to a
> > particular leaf instance, the one with the matching value.  But that
> > idea is not embedded in the Yang semantics of leafref types in any way
> > (other than the output of the deref function), so the fact that there
> > might be more than one matching leaf instance does not matter.
> > 
> > As stated in 9.9.4 and 9.9.5, the lexical representations of its values
> > are the same as those of the referenced nodes.
> > 
> > How is the leafref's value compared to the values of the referenced
> > nodes?  I can see that question getting ugly for the more complex types
> > (e.g., anyxml)
> 
> You can't have a leafref to an anyxml node; just to a leaf or
> leaf-list.
> 
> > which do not have canonical forms.  I suspect the
> > intention is that values are equal if they have the same canonical form
> 
> No, the idea is that they are equal if their *value* is equal,
> regardless of the lexical representation.

There are two types that don't have a canonical form, identityref and
instance-identifier.  It seems that comparisons in XPath expressions
are inexact if the type doesn't have a canonical form (section 6.4).
But if I understand you correctly, the implicit comparisons in leafref
are done based on the abstract values involved, not the lexical
representation.

> > The current ABNF doesn't allow for "+" for joining quoted strings.
> > Also, it doesn't show that \" can be included in a double quoted string
> > to include a literal ", and allows the string contents to continue --
> > the current ABNF "DQUOTE string DQUOTE" matches "abcd\", despite that
> > the latter is not a proper double-quoted string.
> 
> Note that the prose text (within <...>) says "a string that
> matches...".  That string can be any YANG token string, for example
> one of:
> 
>    "hello"
>    "he" + "llo"

If I haven't gotten confused, you're referring to

   string              = < an unquoted string as returned by >
                         < the scanner, that matches the rule >
                         < yang-string >

   yang-string        = *yang-char

   ;; any Unicode or ISO/IEC 10646 character including tab, carriage
   ;; return, and line feed, but excluding the other C0 control
   ;; characters, the surrogate blocks, and the noncharacters.
   yang-char = %x09 / %x0A / %x0D / %x20-D7FF /
                               ; exclude surrogate blocks %xD800-DFFF
              %xE000-FDCF /    ; exclude noncharacters %xFDD0-FDEF
              %xFDF0-FFFD /    ; exclude noncharacters %xFFFE-FFFF
              %x10000-1FFFD /  ; exclude noncharacters %x1FFFE-1FFFF
              %x20000-2FFFD /  ; exclude noncharacters %x2FFFE-2FFFF
              %x30000-3FFFD /  ; exclude noncharacters %x3FFFE-3FFFF
              %x40000-4FFFD /  ; exclude noncharacters %x4FFFE-4FFFF
              %x50000-5FFFD /  ; exclude noncharacters %x5FFFE-5FFFF
              %x60000-6FFFD /  ; exclude noncharacters %x6FFFE-6FFFF
              %x70000-7FFFD /  ; exclude noncharacters %x7FFFE-7FFFF
              %x80000-8FFFD /  ; exclude noncharacters %x8FFFE-8FFFF
              %x90000-9FFFD /  ; exclude noncharacters %x9FFFE-9FFFF
              %xA0000-AFFFD /  ; exclude noncharacters %xAFFFE-AFFFF
              %xB0000-BFFFD /  ; exclude noncharacters %xBFFFE-BFFFF
              %xC0000-CFFFD /  ; exclude noncharacters %xCFFFE-CFFFF
              %xD0000-DFFFD /  ; exclude noncharacters %xDFFFE-DFFFF
              %xE0000-EFFFD /  ; exclude noncharacters %xEFFFE-EFFFF
              %xF0000-FFFFD /  ; exclude noncharacters %xFFFFE-FFFFF
              %x100000-10FFFD  ; exclude noncharacters %x10FFFE-10FFFF

But if that's taken at face value, you can lex as single "string"s not only

    "hello"

and

    "he" + "llo"

but also

    "The MTU of the interface."; myext:c-define "MY_MTU"

Doing that would allow the incorrect lexing of

         leaf mtu {
           type uint32;
           description "The MTU of the interface."; myext:c-define "MY_MTU";
         }

as having a long description (starting with 'The MTU' and ending with
'MY_MTU') and no myext:c-define statement.

What we need is a production that matches "strings possibly combined
with +" and nothing else.  That is, including '"he" + "llo"' but not the
last example.

Dale

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 10:15:24 +0200 (CEST)

Hi,

> Yes, "separate" would work well.

Ok, I'll add this.

> module has meaning separately and can be instantiated seperately.

I'll s/hierarchy/hierarchies/.

> hierarchy" is actually incorrect, and should be "statement hierarchy".

Fixed.

> allows people to write "\r", which is interpreted as backslash-r?

I think there are implementations that reject \r, and I know that
there are implementations that allow \r.

> Yang 1 allows people to use CR alone, and it is not interpreted as a

If the module has a string like this:

   0x27 0x0d 0x27

it is a string with a single CR.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 12:29:59 +0200 (CEST)

Hi,

> So keywords aren't reserved; they can also be used as identifiers.

Yes.

>    braces ("{" or "}"), or comment sequences ("//", "/*", or "*/").

Thanks, better.

> whether it is so relaxed thay any of the types of string tokens can be

No; a keyword must be written w/o quotes, so it is special.

>    line that must be examined to for stripping is first converted into 8

Ok, fixed.

> Or even "is a human-readable cross-reference ...", but either is OK

Ok.

> are done based on the abstract values involved, not the lexical

Yes.

>     "The MTU of the interface."; myext:c-define "MY_MTU"

Why whould this be treated a single string?  Note that line breaks are
not special in YANG, so by the same logic, this example:

  description
    "The MTU of the interface.";
  reference
    "RFC XYZ";

would be scanned as the three tokens:

  description

  "The MTU of the interface.";
  reference
    "RFC XYZ"

  ;  

/martin

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 8 Jun 2016 12:51:06 +0200

> >    braces ("{" or "}"), or comment sequences ("//", "/*", or "*/").

The does the 'is not a keyword' not contratict the example shown
above? My reading of the new suggested text would make me believe
I would have to write the example as follows

  leaf "type" {
      type string;
  }

which is different from the YANG 1.0 syntax rules I think.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 13:16:28 +0200 (CEST)

> >    The argument is a string or a keyword, as defined in Section 6.1.2.

The alternative is to keep the current text.  But it means that the
scanner is context-dependent (logically at least).

IMO it doesn't really matter - this is just a conceptual explanation;
an implementation is free to implement this in whatever clever way it
wants.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 8 Jun 2016 13:48:56 +0200

> an implementation is free to implement this in whatever clever way it

OK. So in combination with the other text we achieve the same result.
But personally, I think it is much clearer for an implementor if the
definition of an unquoted string does not exclude keywords, the risk
of getting inconsistencies later on scares me a bit. I rather would
like to see something like this:

    An unquoted string is any sequence of characters and does not
    contain any space, tab, or newline characters, a single or double
    quote character, a semicolon (";"), braces ("{" or "}"), or
    comment sequences ("//", "/*", or "*/").

    Note that any keyword can legally appear as an unquoted string.

YANG is indeed a bit special here and I believe it helps implementors
to point this out right here where an unquoted string is defined.  And
'the argument' is really a string and not a string or a keyword, which
may lead to confusion or worse wrong interpretations (despite the fact
that one might implement things this way).

/js

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 08 Jun 2016 14:08:24 -0400

>> But if it's true, I seems desirable to provide an explicit warning

It still seems to me to be valuable to warn about this.

> it is a string with a single CR.

I see in the grammar:

   line-break          = CRLF / LF

So you can't write an isolated CR in most of a Yang module.  But am I
correct in understanding that you *can* write an isolated CR inside a
single-quoted string (and presumably, a double-quoted string)?

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 08 Jun 2016 14:34:06 -0400

From: Juergen Schoenwaelder <[email-masked]>
>     Note that any keyword can legally appear as an unquoted string.

That text seems to be quite clear to me.  As Juergen notes, it directly
points out to the implementor an important fact about lexing Yang.
(Similarly, some programming languages allow keywords to be used as
identifiers, they're not reserved.)

> No; a keyword must be written w/o quotes, so it is special.

OK, that answers one '?'.  But there are effectively two that remain:

Is abc   valid as a string?
Is "abc" valid as an identifier?

I *think* the answer is No for both of those, but I can't put my finger
on the rules that make that definite.

>> are done based on the abstract values involved, not the lexical

I'm willing to take that as understood.

>   "The MTU of the interface.";

Actually, I've gotten completely lost in regard to how the grammar
specifies how strings can be written.  Let me take a single example.
There are these productions:

   organization-stmt   = organization-keyword sep string stmtend

   yang-string        = *yang-char

(I won't quote the production for yang-char.)  As written, this seems to
say that in an organization statement, there is the organization
keyword, a sep, a string, and then a stmtend.  But the text above says
that a string is "an unquoted string as returned by the scanner, that
matches the rule yang-string".  But looking at the example:

       organization "Example Inc.";

I see that the string involved is quoted, which doesn't seem to be
allowed by the production for "string", because the production says "an
unquoted string".

Now I suspect that what the production for "string" really wants to be
is "a quoted or unquoted string whose *value* matches the rule
yang-string".  That is, what is *written* isn't allowed to be any
*yang-char, but the denoted value must be *yang-char.  (Some of the
other productions seem to have the same issue.)

Another point which is confusing me is how quoted strings are specified
by the grammar.  The only appearance of DQUOTE is in

   quoted-string       = (DQUOTE string DQUOTE) / (SQUOTE string SQUOTE)

and the only appearance of quoted-string is in

   key-predicate-expr  = node-identifier *WSP "=" *WSP quoted-string

   leaf-list-predicate-expr = "." *WSP "=" *WSP quoted-string

which leaves me wondering how quoted strings are allowed elsewhere in
the language.

Dale

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 22:45:52 +0200 (CEST)

> It still seems to me to be valuable to warn about this.

Do you have an suggestion for what to write?  These kinds of
multi-line strings are almost always 'description' statements,
used for human consumption.  It is not clear to me what the warning
would be.

Yes.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 22:59:59 +0200 (CEST)

> identifiers, they're not reserved.)

Ok, let's use Juergen's proposed text.

Yes.

Yes.

All YANG statements have the same structure:

  statement = keyword [argument] (";" / "{" *statement "}")

All arguments are strings.  Strings can be unquoted or quoted (and
possibly concatenated).

> specifies how strings can be written.

The grammar doesn't specify how strings can be written.  This is
defined in the section "Lexical Tokenization":

   unquoted
   quoted
   quoted and concatenated

> Let me take a single example.
> that a string is "an unquoted string as returned by the scanner

This means that in all these examples, the resulting string is
the same:

   hello
   "hello"
   'hello'
   'he' + "llo"

and the resulting string is a string with 5 characters.

>, that
> *yang-char, but the denoted value must be *yang-char.

Yes, but it is more than quoted or unquoted; it is also concatenation,
escape char substition and whitespace trimming.

> (Some of the

Note that these are quoted strings *after* the scanner's processing of
the input, so e.g., you might have:

  path '/foo[bar="hi"]';

/martin

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 09 Jun 2016 17:08:44 -0400

>> single-quoted string (and presumably, a double-quoted string)?

I think all of this could be summarized well by putting a paragraph that
tells everything about line ends in Yang at the end of section 6.
Section 6 already deals with the valid characaters for Yang, how they're
encoded, etc.

    Lines in a YANG module may end with a carriage return-line feed
    combination or with a line feed alone.  A carriage return that is not
    followed by a line feed may only appear inside a quoted string.  Note
    that carriage returns and line feeds that appear inside quoted strings
    become part of the value of the string without modification; the value
    of a multi-line quoted string contains the same form of line ends as
    those lines of the YANG module.

In a way, that doesn't add much to the specification.  As I think you
said, "Line ends aren't special in Yang."  But it does spell out exactly
when CR and LF can be used, and it warns readers against misleading
analogies from other programming languages (where line ends usually are
special).

Dale

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 09 Jun 2016 23:17:59 +0200 (CEST)

[email-masked] (Dale R. Worley) wrote:
> In a way, that doesn't add much to the specification.

Right, it provides additional clarification.  Thank you for this text!

/martin

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 09 Jun 2016 18:03:50 -0400

At this point, I think there are some aspects of the use of strings and
identifiers that aren't completely specified, but I'd have to check the
current text to know for certain.  However, that's a minor matter.

The issue that concerns me is that the ABNF doesn't specify what is
allowed as a string.  I'm used to programming language definitions,
where the grammar is specified quite rigidly, to the point that the ABNF
can be input to a parser generator.  In this document, the ABNF is quite
complete except for a specification of strings.  On the other hand, the
text description of strings seems to be sufficient for an implementer,
so we don't actually need to provide ABNF.  My strong preference is to
provide a complete ABNF, as is the norm for programming languages.

The following is a complete ABNF for Yang strings.  Of course, it's a
bit complicated, because the definition of strings in Yang actually is a
bit complicated.

   string              = unquoted-string / quoted-string

   unquoted-string     = *unquoted-item ( unquoted-item /
                                          "/" /
                                          "*" *"*" )
                         ;; a sequence of one or more characters from
                         ;; ordinary-char / "/" / "*", not containing
                         ;; "//", "/*", or "*/"

   unquoted-item       = ordinary-char /
                         "/" ordinary-char /
                         "*" *"*" ordinary-char

   ordinary-char       = < any character matching yang-char, except >
                         < space, tab, newline, carriage return,    >
                         < semicolon, left brace, right brace,      >
                         < slash, and asterisk                      >

*** Hmmm, is an unaccompanied CR allowed in an unquoted string?  If
so, "ordinary-char" and my previous text regarding line breaks need
to be amended.

   quoted-string      = ( single-quoted-string / double-quoted-string )
                        *( optsep "+" optsep
                           ( single-quoted-string / double-quoted-string ) )

(I think you said that there can be whitespace around + but not comments.)

   single-quoted-string = SQUOTE *sq-char SQUOTE

   sq-char            = < any character matching yang-char, except >
   		        < SQUOTE                                   >

   double-quoted-string = DQUOTE *dq-item DQUOTE

   dq-item            = dq-char /
   		      	"\n" /
			"\t" /
			"\" DQUOTE /
			"\\"

   dq-char            = < any character matching yang-char, except >
   		        < DQUOTE and backslash                     >

(The existing production for yang-string is removed.)

   [continuing as before]

Dale

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 10 Jun 2016 09:35:29 +0200

I think we should consider this for a future version of YANG but not
now. I am not aware what implementors had problems to implement YANG
strings in YANG 1 or YANG 1.1 and we are at a point in the process
where I like to not run the risk to make bigger changes to the
specification that may turn out to introduce incompabilities.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 10 Jun 2016 10:00:53 +0200 (CEST)

> specification that may turn out to introduce incompabilities.

I agree.

> > so, "ordinary-char" and my previous text regarding line breaks need

To clarify this, I suggest:

OLD:

  An unquoted string is any sequence of characters that does not
  contain any space, tab or newline characters, a single or double
  quote character, a semicolon (";"), braces ("{" or "}"), or comment
  sequences ("//", "/*", or "*/").

NEW:

  An unquoted string is any sequence of characters that does not
  contain any space, tab, carrige return, or line feed characters, a
  single or double quote character, a semicolon (";"), braces ("{" or
  "}"), or comment sequences ("//", "/*", or "*/").

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 10 Jun 2016 10:13:33 +0200

>   "}"), or comment sequences ("//", "/*", or "*/").

This looks good to me.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 10 Jun 2016 10:38:28 +0200

> On 10 Jun 2016, at 10:00, Martin Bjorklund <[email-masked]> wrote:
> 

s/carrige/carriage/

Lada

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: [email-masked] (Dale R. Worley)
Date: Mon, 13 Jun 2016 11:54:55 -0400

> specification that may turn out to introduce incompabilities.

I've written an issue about it
(https://github.com/netmod-wg/yang-next/issues/6), so it can be
considered for the next revision.

Dale