Subject: Re: [netmod] leafref value space and constraint

From: [email-masked] (Dale R. Worley)
Date: Thu, 02 Jun 2016 16:14:56 -0400

Martin Bjorklund <mbj at tail-f.com> wrote:
> Also, the text needs to mention leaf-lists.  This gives:
> 
>   The leafref type is restricted to the value space of some leaf or
>   leaf-list node in the schema tree and optionally further restricted
>   by corresponding instance nodes in the data tree.  The "path"
>   substatement (Section 9.9.2) is used to identify the referred leaf
>   or leaf-list node in the schema tree. The value space of the
>   referring node is the value space of the referred node.
> 
>   If the "require-instance" property is "true", there MUST exist an
>   node in the data tree, or a node with a default value in use (see
>   Section 7.6.1 and Section 7.7.2), of the referred schema tree leaf
>   or leaf-list node with the same value as the leafref value in a
>   valid data tree.
> 
>   If the referring node represents configuration data, and the
>   "require-instance" property (Section 9.9.3) is "true", the referred
>   node MUST also represent configuration.

How about:

    The leafref type contains an XPath expression in the "path"
    substatement.  Due to the restrictions on the expression, all
    instances in the result set of the expression evaluated on all
    possible data tree are instances of one single data node, called the
    "referred node".

    The leafref type has the value space of the type of the referred
    node.
                              
    If the "require-instance" property is "true", the leafref type also
    imposes a constraint on any node of its type:  the value must be the
    same as that of some instance in the data tree, or an instance with
    a default value in use, selected by the XPath expression.  Such a
    constraint is enforced according to the rules in Section 8.

    If the referring node represents configuration data, and the
    "require-instance" property (Section 9.9.3) is "true", the referred
    node MUST also represent configuration.

I started with a discussion of the XPath expression to state explicitly
that all selected instances must be instances of one data node, and used
that to define "referred node".  With that clarified, it's easier to
state the conditions that we care about.

I want to clearly separate the question of value space (which is
enforced roughly any time XML using the module is parsed) from the
constraint (which is enforced only at the times specified in section 8).

I do exploit the convention that default values cause instance nodes to
exist in the data tree in a sort of virtual sense, in that the XPath
expression can return them in its data set.  But I think that is
consistent with the philosophy of Yang.  Does that need to be stated
more explicitly?

Also, in the lexicon:

   o  anydata: A data node that can contain an unknown set of nodes that
      can be modelled by YANG, except anyxml.

   o  anyxml: A data node that can contain an unknown chunk of XML data.

Both definitions should start with "a data node *whose instances* can
contain..."

Dale

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 6 Jun 2016 19:08:48 +0200

>     possible data tree are instances of one single data node, called the

I think the first sentence is potentially misleading (the "path"
expression does not contain arbitrary XPath) and the second sentence
is really hard to parse - if it makes sense at all the way it is
written.
 
> state the conditions that we care about.

Martin's original text says 'schema node' so there is not confusion
that I see. The referred node is in the schema tree, not in the data
tree.

> consistent with the philosophy of Yang.  Does that need to be stated
 
I think we should go with Martin's text and close this issue.
 
> Both definitions should start with "a data node *whose instances* can

I would leave this is Martin's discretion (since he knows best whether
this causes inconsistencies with other wordings in the document).

Martin, how close are we to have a final version addressing all the
review comments we received during the journey through the IESG?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: [email-masked] (Dale R. Worley)
Date: Mon, 06 Jun 2016 14:06:57 -0400

A difficulty I have with the current wording is that it doesn't point
out the crucial fact about leafref that the XPath expression can only
select elements that are instantiations of one particular data node.  I
don't know XPath, but it seems that that is not a general property of
XPath expressions, you seem to be able to write XPath expressions that
select heterogenous groups of elements.  So it's worth pointing out that
the allowed XPath expressions can't do that, and that is because of the
restriction that the expression must match path-arg.

Perhaps my problem is that XPath is almost always used in this style,
and so the limitations in leafref aren't anything unusual.

Dale

---

From: Andy Bierman <[email-masked]>
Date: Mon, 6 Jun 2016 11:14:57 -0700

> and so the limitations in leafref aren't anything unusual.
I can say that the implementation details are much easier than the
convoluted
text might imply.

1) the value MUST be valid for the final "type" statement.
The server validates that the value is valid for the type of the pointed-at
leaf.
(Or final pointed-at leaf if it is also a leafref).

2) If require-instance is true then the value MUST also be in use by
the first pointed-at leaf. i.e., the value is valid for the data type and
there exists an instance of the pointed-at leaf with the same canonical
value

> Dale

Andy

>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 6 Jun 2016 20:44:11 +0200

> and so the limitations in leafref aren't anything unusual.

Does section 9.9.2, which describes the path statement, not describe
this is sufficient detail? Note also that 9.9.2 is a subsection of
9.9, which describes the leafref type.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 07 Jun 2016 11:25:24 +0200

> restriction that the expression must match path-arg.

Right, this is the slightly hand-waving part that I also objected to. An
XPath expression in a leafref's "path" statement indeed selects a node
set from an instance data tree. The node seet can be empty, but if it is
non-empty, all its members necessarily have the same type, which is
defined in a certain "leaf" schema node.

The relationship is relatively straightforward but difficult to explain
concisely. One basically has to follow the steps in the XPath
expression, ignore all path-predicates, and skip all schema nodes that
aren't data nodes (i.e., "choice" and "case" nodes).

> and so the limitations in leafref aren't anything unusual.

Actually, no, XPath expressions always have to be evaluated in the context of a
specific instance XML document, they cannot directly address schema
nodes. XPath 1.0 spec says: "XPath is a language for addressing parts of
an XML document, ...".

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 07 Jun 2016 14:39:33 +0200 (CEST)

> review comments we received during the journey through the IESG?

We're shrinking the list of open issues with every iteration of
emails.  I will go through the latest comments from the Gen-ART and
ops-dir reviews and reply to the list.  Maybe we can be done by the
end of this week.

/martin

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 07 Jun 2016 11:26:03 -0400

> aren't data nodes (i.e., "choice" and "case" nodes).

Yes, it's difficult to explain, though once one gets the idea, it's
straightforward enough.  The problem I have is that it's not pointed out
explicitly anywhere.  (E.g., Juergen suggests section 9.9.2, but I don't
see it there.)

The crucial points seem to be:

- The XPath expression is limited by the syntax "path-arg" and the rules
  in 9.9.2.

- Because of those restrictions, there exists one data node in the
  schema such that:  evalutating the expression for any leaf or
  leaf-list node in any data tree returns a set of nodes, all of which
  are instances of that one schema node.

- The type of that schema node is the base type of the leafref.

- Once you learn this, it's easy to see, for any particular
  leaf/leaf-list node and path expression in a module, which schema node
  is the one.  But it's rather hard to describe that process.

Actually, there's an ugly question:  If the path expression references
an element name that doesn't exist in the module.  Perhaps there are
rules in XPath that prevent it, but it seems to me that you could write

     leaf mgmt-interface {
       mandatory false;
       type leafref {
         path "../interface/name";
         require-instance true;
       }
     }

when the current module doesn't have a "name" child defined for
"interface".  As long as a data tree didn't contain a mgmt-interface
elememt, the constraint would not be violated.

But a later revision of the module, the "name" child could be added,
allowing data trees to contain mgmt-interface elements, because data
trees could now have "name" elements.

The point being that when you're figuring out which schema node is the
base type for the leafref, that schema node has to exist so the base
type can be extracted.  But there's no direct statement of that as a
requirement for path validity.

Dale

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 7 Jun 2016 17:49:58 +0200

> On 07 Jun 2016, at 17:26, Dale R. Worley <[email-masked]> wrote:
> 

Yes, this is correct.

> rules in XPath that prevent it, but it seems to me that you could write

Actually, no, such an XPath is perfectly okay, it will just evaluate to an empty node set, exactly as if the schema node exists but there is no instance in the data tree - XPath just doesn't care about schemas.

> elememt, the constraint would not be violated.

Hmm, I think you are right. Practically speaking, I would expect all YANG module compilers to complain about nonexistent schema node, but it follows neither from the old nor the new text.

> requirement for path validity.

I guess a honest way out of this problem would be to describe how to get the corresponding schema node from the path expression, and state that it must always exist. This can't be done in one line or two though.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jun 2016 17:52:58 +0200

> requirement for path validity.

As WG chair, I am getting a bit nervous. We have to wrap things up and
we need deliver the specification - it is past WG last call, it is
past IETF last call, it is past IESG approval (kind of). We can likely
endlessly continue to search for possible ways to misunderstand the
specification but lets remember the aphorism 'perfect is the enemy of
good'.

I leave it to Martin's discretion to decide whether he wants to add
'The referred leaf or leaf-list node in the schema tree MUST exist.'
to the new text but I think the time has come to stop searching for
new possible ways to misunderstand the specification.

I like to see all issues closed by the end of the week a new revision
with all changes posted. People can then check the edits to be sure
nothing broke and then the document should move via Benoit into the
RFC editor queue by the end of the following week or Monday June 20th.

/js

---

From: Andy Bierman <[email-masked]>
Date: Tue, 7 Jun 2016 09:08:56 -0700

> new possible ways to misunderstand the specification.

Doesn't the text above exclude a default leaf or leaf-list from the
value space?  That would be incorrect.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 Jun 2016 18:38:31 +0200

> value space?  That would be incorrect.

It says 'in the schema tree' so I think there is no problem. We were
discussing the corner case where the path argument does not point to a
schema node at all. Pyang already reports this as an error (bar in the
path of foo [...] is not found).

/js

---

From: Andy Bierman <[email-masked]>
Date: Tue, 7 Jun 2016 09:45:36 -0700

On Tue, Jun 7, 2016 at 9:38 AM, Juergen Schoenwaelder <
[email-masked]> wrote:

> > > > >> select elements that are instantiations of one particular data
> node.
> > > I
> > > > >> select heterogenous groups of elements.  So it's worth pointing
> out
> > > that
> > > > >> the allowed XPath expressions can't do that, and that is because
> of
> > > the
> > > > > Right, this is the slightly hand-waving part that I also objected
> to.
> > > An
> > > > > set from an instance data tree. The node seet can be empty, but if
> it
> > > is
I thought the original issue was that the ABNF does not support
position-based
instance identifiers.  In the rare event that the i-i pointed to a
config=false
list entry (with no keys defined) then its position needs to be used.

I thought it was clear that the path-stmt has to point at a schema node
that can be resolved in the module context using the import-stmts.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 09:48:12 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> instance identifiers.

This is supported even in RFC 6020.

/martin

> In the rare event that the i-i pointed to a
> config=false

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jun 2016 09:51:15 +0200 (CEST)

> 'The referred leaf or leaf-list node in the schema tree MUST exist.'

The text for "path" already says:

  It takes as an argument a
  string that MUST refer to a leaf or leaf-list node.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 8 Jun 2016 10:10:28 +0200

> On 08 Jun 2016, at 09:51, Martin Bjorklund <[email-masked]> wrote:
> 

It's unclear what "refer" means. If it's XPath, it can only refer to an instance in a data tree, not in the schema.

Lada

---

From: Per Hedeland <[email-masked]>
Date: Wed, 8 Jun 2016 10:31:09 +0200

> This is supported even in RFC 6020.

But not for the leafref 'path' argument, right? I'm not sure how
instance-identifier values are relevant in this particular discussion...

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 8 Jun 2016 11:13:32 +0200

> On 07 Jun 2016, at 18:45, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> On Tue, Jun 7, 2016 at 9:38 AM, Juergen Schoenwaelder <[email-masked]> wrote:
> I thought the original issue was that the ABNF does not support position-based
> instance identifiers.  In the rare event that the i-i pointed to a config=false

A more significant change was the addition of the "require-instance" substatement to the leafref type.

> that can be resolved in the module context using the import-stmts.

An example:

  container top {
    leaf foo {
      type string;
    }
    container subtop {
      choice choi {
        leaf foo {
          type uint8;
        }
        case barcase {
          leaf bar {
            type leafref {
              path "../../foo";
            }
          }
        }
      }
    }
  }

Which of the two "foo" leaf nodes does the path argument point at? If the path is interpreted in the schema tree, it may appear it is the one inside the choice. Yes, we all know it points at the other one, but will all readers of the spec come to the same conclusion?

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 08 Jun 2016 13:38:29 +0200

OK, let's bite the bullet:

OLD

   The "path" statement, which is a substatement to the "type"
   statement, MUST be present if the type is "leafref".  It takes as an
   argument a string that MUST refer to a leaf or leaf-list node.

   The syntax for a path argument is a subset of the XPath abbreviated
   syntax.  Predicates are used only for constraining the values for the
   key nodes for list entries.  Each predicate consists of exactly one
   equality test per key, and multiple adjacent predicates MAY be
   present if a list has multiple keys.  The syntax is formally defined
   by the rule "path-arg" in Section 14.

NEW

   The "path" statement, which is a substatement to the "type"
   statement, MUST be present if the type is "leafref". The syntax for
   its argument is formally defined by the rule "path-arg" in
   Section 14.

   The "path" statement is used in two ways:

   1. It refers to another "leaf" node that is obtained by treating the
      "path" argument as a path in the schema tree. If the argument
      begins with the slash character ("/"), i.e., corresponds to the
      ABNF rule "absolute-path", the path starts at the root of the
      schema tree, otherwise it starts at the current "leaf" node with
      the "leafref" type.

      The steps along this path are performed by interpreting each
      of the slash-separated components of the "path" argument as
      follows:

      o If the component is the double-dot string (".."), go to the
        closest schema tree ancestor that is a data node or schema root.

      o Otherwise, the component contains a data node identifier and an
        optional path predicate enclosed in square brackets. Go to the
        data child (see below) whose name is the data node identifier.

        A data child of data node N is a descendant node in the schema
        tree for which node N is the closest ancestor that is a data
        node.

        The path predicate, if present, is ignored.

      The "leaf" node to which the "path" argument refers MUST exist in
      the schema tree.

    2. If the "require-instance" property (Section 9.9.3) is "true", the
       "path" argument is evaluated as an XPath expression once for each
       instance of the "leaf" with the "leafref" type. The result of
       every such evaluation MUST be a non-empty node set.

   Path predicates are used only for specifying the values for the
   key nodes for list entries.  Each predicate consists of exactly one
   equality test per key, and multiple adjacent predicates MAY be
   present if a list has multiple keys.

Lada

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 08 Jun 2016 10:59:51 -0400

[email-masked] (Dale R. Worley) writes:
> [...]

Interestingly, there's a sort-of dependency on the fact that
grouping/uses can't be recursive.  If I could define a general binary
tree data structure:

    grouping tree-grouping {
      container left-child {
        uses tree-grouping;
      }
      container right-child {
        uses tree-grouping;
      }
      leaf data1 {
        type string;
      }
      leaf data2 {
        type leafref {
          path "../left-child/data1";
        }
      }
    }

    container a {
      use tree-grouping;
      container left-child {
        leaf data1 {
          type integer;
        }
      }
    }

statically validating the leafref would take a clever algorithm, because
the parent element of a data2 element can be a left-child container, a
right-child container, or an 'a' container.

But since groupings cannot be instantiated recursively, the validator
can "macro expand" all 'uses' statements and then validate every
instance of the leaf data2; for each instance of data2, the parent
element is unique, making interpretation of the path simple.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 08 Jun 2016 11:22:37 -0400

>   string that MUST refer to a leaf or leaf-list node.

Given what the text must mean, that's sufficient for me.

Ladislav Lhotka <[email-masked]> writes:
> Which of the two "foo" leaf nodes does the path argument point at? If
> the path is interpreted in the schema tree, it may appear it is the
> one inside the choice. Yes, we all know it points at the other one,
> but will all readers of the spec come to the same conclusion?

Well, in 9.9.2 it says:

   The "path" XPath expression is conceptually evaluated in the
   following context, in addition to the definition in Section 6.4.1:

   o  If the "path" statement is defined within a typedef, the context
      node is the leaf or leaf-list node in the data tree that
      references the typedef.

   o  Otherwise, the context node is the node in the data tree for which
      the "path" statement is defined.

What's really going on is that the XPath expression will always be
evaluated in the context of a particular element of a particular data
tree, but that it can be shown to be always valid by doing a "generic
evaluation" of the expression looking at the schema tree -- the schema
tree is a generic description of all possible data trees.  Of course
there are various complications because not all nodes in the schema tree
are represented in the XML tree.

In mathematics, abstracting an operation in a particular structure to a
similar operation in a "generic" structure is called "lifting", and
using that term, it's easy to notify the reader what you're doing
without spelling out the details.  What's difficult here is that it's
"obvious" what the text has to mean, but we have no simple vocabulary
for pointing that out, and the reader who is not careful might not
realize everything that is going on.

At this point, I'm willing to ignore the issue.  There doesn't seem to
be a simple way to explain the critical issue.  And if the reader
attempts to work out the details of Yang processing, he eventually has
to come to the correct conclusion; there is only one self-consistent
interpretation.

Dale

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 9 Jun 2016 13:02:18 +0200

> On 08 Jun 2016, at 17:22, Dale R. Worley <[email-masked]> wrote:
> 

Then it's fine with me, too.

Lada