Subject: [netmod] YANG1.1 new issue: add 'mandatory' to container

From: chong feng <[email-masked]>
Date: Mon, 21 Apr 2014 20:25:34 +0800

Hi all,
    I suggest add 'mandatory' to container to solve the problem listed
below:
    container a {
          leaf b;
          leaf c;
    }

    user can set b or set c or set b and c,but at least one is assigned.

   So, I suggest add 'mandatory' to container to express this situation. If
a container is mandatory, at least one sub path of this container should be
selected when the instance associated with this container is created. The
container with presence must not be set "mandatory true".

---

From: Andy Bierman <[email-masked]>
Date: Mon, 21 Apr 2014 06:13:19 -0700

>     user can set b or set c or set b and c,but at least one is assigned.

You can use must-stmt:

   container a {
      must "b or c";
      leaf b { ...}
      leaf c {...}
   }

Top-level mandatory nodes are not allowed in IETF YANG modules,
so in order to make 'a' mandatory, it has to be a child node of something:

  container top {
     must "a";
     container a { ... }
  }

>    So, I suggest add 'mandatory' to container to express this situation.
> If a container is mandatory, at least one sub path of this container should
> be selected when the instance associated with this container is created.
> The container with presence must not be set "mandatory true".
>
>
An NP-container has no semantics.  It is just a way of collecting nodes
under a common parent node.  So mandatory is not relevant for NP-container.
NP-container is a corner-case node because it is transparent to default and
mandatory,
meaning child nodes can make the container a default or mandatory already.

 container a {
   must "b or c";   // always true
   leaf b { type int32; default 42; }
   leaf c { type string; }
   leaf d { mandatory true; type int32; }
 }

The node /a/b always exists wrt/ XPath evaluation.
Container 'a' is mandatory because leaf 'd' is mandatory.

Andy

---

From: chong feng <[email-masked]>
Date: Tue, 22 Apr 2014 00:57:58 +0800

Hi andy,
   I don't think 'must' is a good solution.
   For example:
   The hierarchy of data def is below:
   module test {
        container a {
              container b {
                    container c {
                          container d {
                                 leaf e;
                                 leaf f;
                          }
                    }
              }
        }
  }

if use 'must' ,it should be :
   module test {
        container a {
              must b;
              container b {
                    must c;
                    container c {
                          must d;
                          container d {
                                 must "e or f";
                                 leaf e;
                                 leaf f;
                          }
                    }
              }
        }
  }

but container a can be not selected, because 'must' expression only
be evaluated when container a is exists.

if use 'mandatory', it should be:
   module test {
        container a {
              container b {
                    container c {
                          container d {
                                 mandatory true;
                                 leaf e;
                                 leaf f;
                          }
                    }
              }
        }
  }

container a must be selected when the data is created.

If a NP-container is just a path node, then the container 'd' will be  not
a common NP-container, It's also represent a logical
relation(one or more sub path must be selected).

Should we add a new datadef stmt? for example, multi-choice?

for example:
   module test {
        container a {
              container b {
                    container c {
                          multi-choice d {
                                 mandatory true;
                                 leaf e;
                                 leaf f;
                          }
                    }
              }
        }
  }

> On Mon, Apr 21, 2014 at 5:25 AM, chong feng <[email-masked]>wrote:
>
>> Hi all,
> NP-container is a corner-case node because it is transparent to default
> and mandatory,

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 21 Apr 2014 19:57:40 +0200

On 21 Apr 2014, at 15:13, Andy Bierman <[email-masked]> wrote:

>     I suggest add 'mandatory' to container to solve the problem listed below:
>     container a {
>           leaf b;
>           leaf c;
>     }
> 

This doesn’t work. If container “a" doesn’t exist in the data tree, the must constraint doesn’t apply (second paragraph in sec. 7.5.3).

Lada

>    So, I suggest add 'mandatory' to container to express this situation. If a container is mandatory, at least one sub path of this container should be selected when the instance associated with this container is created. The container with presence must not be set "mandatory true".
> 
> 
> NP-container is a corner-case node because it is transparent to default and mandatory,
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 21 Apr 2014 12:28:35 -0700

On Mon, Apr 21, 2014 at 10:57 AM, Ladislav Lhotka <[email-masked]> wrote:

> This doesn't work. If container "a" doesn't exist in the data tree, the
> must constraint doesn't apply (second paragraph in sec. 7.5.3).
>
>

I understand that.
Top-level mandatory nodes are not allowed in IETF YANG modules
because they are implementation-dependent.  As soon as the module is
loaded, the configuration is invalid because the mandatory node
is missing.

I do not agree NP-containers need the mandatory-stmt.

Andy

---

From: Per Hedeland <[email-masked]>
Date: Mon, 21 Apr 2014 21:52:26 +0200

On 2014-04-21 19:57, Ladislav Lhotka wrote:
> This doesnt work. If container a" doesnt exist in the data tree, the must constraint doesnt apply (second paragraph in sec. 7.5.3).

You seem to be assuming that the suggested "mandatory" on a container
should be radically different from the existing "mandatory" on leafs and
choices - why? If container a doesn't exist, "mandatory" on e.g. leaf b
has no effect - nor should the suggested "mandatory" on container a,
meaning "b and/or c must be set", have any effect. Consider

   container p {
     presence "foo";
     container a {
       mandatory true;  // new suggestion
       leaf b {...}
       leaf c {...}
     }
   }

If p doesn't exist, it would be completely unreasonable for the
"mandatory" on a to force p into existence - and without that, obviously
neither b nor c could be set. At least it would be completely at odds
with the semantics of the existing "mandatory" statement.

Bottom line, "must" works just as well/bad as the new suggestion.

--Per

---

From: Per Hedeland <[email-masked]>
Date: Mon, 21 Apr 2014 22:11:38 +0200

> Bottom line, "must" works just as well/bad as the new suggestion.

Hm, or are you saying that, with

   container p {
     presence "foo";
     container a {
       must "b or c";
       leaf b {...}
       leaf c {...}
     }
   }

- the "must" wouldn't apply even if p existed? In that case there
probably needs to be some clarification of what it means for a
np-container to "exist" for the purpose of "must" evaluation. Our
implementation certainly considers a to "exist" if p exists (regardless
of whether either of b and c exists) - and any other interpretation
seems bizarre to me, but maybe it isn't 110% clear from 6020...

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 21 Apr 2014 22:36:32 +0200

> - the "must" wouldn't apply even if p existed? In that case there

Sure, if “a” doesn’t exist.

> seems bizarre to me, but maybe it isn't 110% clear from 6020…

Bizarre? Sec. 7.5.8:

   If a container does not have a "presence" statement and the last
   child node is deleted, the NETCONF server MAY delete the container.

Lada

---

From: Per Hedeland <[email-masked]>
Date: Mon, 21 Apr 2014 23:08:07 +0200

On 2014-04-21 22:36, Ladislav Lhotka wrote:
> Sure, if a doesnt exist.
>> seems bizarre to me, but maybe it isn't 110% clear from 6020&
> 
> Bizarre? Sec. 7.5.8:
> 

The point is - and this *is* expressed in 6020 (7.5.1) - that the
"existence" of a np-container has no semantics. What you are saying (and
6020 doesn't really contradict you) is that

  container p {
    presence "foo";
    container a {
      leaf b {
        type ...;
        mandatory true;
      }
    }
  }

and

  container p {
    presence "foo";
    container a {
      must "b";
      leaf b {
        type ...;
      }
    }
  }

mean radically different things, and actually that the semantics of the
latter is basically undefined, since it would depend on whether
container a "happened to" exist, even though it can neither be created
nor deleted explicitly, and its existence isn't supposed to have any
semantics.

I'm "pretty" sure this is not the intention of the authors. I seems what
is needed is a clarification (in 7.5.3) of when the "must" statements of
a np-container apply, similar to what is done for "mandatory" in 7.6.5.

--Per

---

From: Andy Bierman <[email-masked]>
Date: Mon, 21 Apr 2014 16:11:12 -0700

On Mon, Apr 21, 2014 at 2:08 PM, Per Hedeland <[email-masked]> wrote:

> >>>> This doesn t work. If container  a" doesn t exist in the data tree,
> the must constraint doesn t apply (second paragraph in sec. 7.5.3).
> >>>
> > Sure, if  a  doesn t exist.
> >
I am confused.
These examples are different.
In the first container 'p' definition, container 'a' is mandatory because
NP-containers with mandatory child nodes are mandatory.
In the 2nd definition, leaf 'b' must be present if its parent container 'a'
is present, but the 'a' container is not itself mandatory.

There is a hack (and also a demonstration of why this request may be
reasonable after all):

   choice mandatory-np-container {
      mandatory true;
      container np-container { ... }
    }

I just made a mandatory NP-container.
Is this workaround good enough or should 'mandatory true' be allowed
directly in the NP container?

--Per

Andy

---

From: chong feng <[email-masked]>
Date: Tue, 22 Apr 2014 07:27:20 +0800

2014-04-22 7:11 GMT+08:00 Andy Bierman <[email-masked]>:

> I think add 'mandatory true' directly is simple on this situation.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 21 Apr 2014 16:37:29 -0700

>> I think add 'mandatory true' directly is simple on this situation.

I think you are right.

(Martin, add it to the list please).

BTW, does sec 10 allow any data node
to be converted into a choice-of-1 in a future revision?

   o  Any set of data definition nodes may be replaced with another set
      of syntactically and semantically equivalent nodes.  For example,
      a set of leafs may be replaced by a uses of a grouping with the
      same leafs.

E.g.:

    container foo { ... }

changed to

    choice foo {
       container foo { ... }
     }

Is this legal because of the clause in sec. 10 cited above?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 22 Apr 2014 08:46:22 +0200 (CEST)

> > a np-container apply, similar to what is done for "mandatory" in 7.6.5.

Yes, I agree with this.

Since the NP-container itself has no semantics, so it cannot change
the meaning of must or mandatory leafs.

Thus, the requested functionality can be done today with "must".

> directly in the NP container?

Having a "mandatory NP container" in itself makes no sense.  Since the
NP container doesn't have any semantics, what does it mean to make it
mandatory?

I don't think we should allow "mandatory" in an NP container just b/c
this hack exists.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 22 Apr 2014 08:53:18 +0200 (CEST)

> (Martin, add it to the list please).

Done.  Y38.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 22 Apr 2014 00:12:41 -0700

On Mon, Apr 21, 2014 at 11:46 PM, Martin Bjorklund <[email-masked]> wrote:

> > > >>>>> On Mon, Apr 21, 2014 at 5:25 AM, chong feng <
> [email-masked]>
> > > wrote:
> > > >>>>> Hi all,
> > > >>> should be radically different from the existing "mandatory" on
> leafs
> > > and
not so sure about this transparency argument

 container-stmt      = container-keyword sep identifier-arg-str optsep
                         (";" /
                          "{" stmtsep
                              ;; these stmts can appear in any order
                              [when-stmt stmtsep]
                              *(if-feature-stmt stmtsep)
                              *(must-stmt stmtsep)
                              [presence-stmt stmtsep]
                              [config-stmt stmtsep]
                              [status-stmt stmtsep]
                              [description-stmt stmtsep]
                              [reference-stmt stmtsep]
                              *((typedef-stmt /
                                 grouping-stmt) stmtsep)
                              *(data-def-stmt stmtsep)
                          "}")

when-stmt and if-feature-stmt make all the child nodes conditional

must-stmt could be viewed in the same way -- convenient way to apply

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 22 Apr 2014 09:17:01 +0200 (CEST)

> when-stmt and if-feature-stmt make all the child nodes conditional

They do that also if you move them from the NP-container into the
subnodes.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 22 Apr 2014 14:29:33 +0200

> "existence" of a np-container has no semantics. What you are saying (and

They may not have semantics from YANG's viewpoint but they are still nodes in the data tree that either exist or not, and as such influence the context for XPath evaluation.

> mean radically different things, and actually that the semantics of the

They do mean different things! The "mandatory" statement is a schema constraint, and sec. 3.1 then explicitly states that "a" is a mandatory node. In contrast, "must" is a semantic constraint that (as any XPath expression) has to be evaluated in the well-defined context of a concrete XML instance document. And it is then significant whether the expression's context node exists or not.

It has been a recurring issue in the previous discussions about "must" and especially "when" that the meaning of XPath expressions is being sort of shifted to the schema, under the assumption that an implementation can (temporarily) insert dummy nodes here and there in the instance document, and that the gentle implementor will figure out the intentions of the data modeller. I think this is a wrong approach that can lead to interoperability problems and race conditions.

> a np-container apply, similar to what is done for "mandatory" in 7.6.5.

To support your view, what's essentially needed is to state that NP-containers are always part of the default contents.

Lada

---

From: chong feng <[email-masked]>
Date: Tue, 22 Apr 2014 23:28:51 +0800

2014-04-22 14:46 GMT+08:00 Martin Bjorklund <[email-masked]>:

> If a NP-container has a mandatory subnode, it means this NP-container is a
actually mandatory node ,
 although  it has no 'mandatory' sub stmt.

The problem that i want to solve is the union of two or more sub nodes are
mandatory,so,
the NP-container is actually mandatory. but there is no method to express
this situation.

The way to define 'must' expression under np-container  can not make it be
mandatory.

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 22 Apr 2014 18:08:31 +0200 (CEST)

> The problem that i want to solve is the union of two or more sub nodes are

Then it feels wrong to mark the container as mandatory.  The must
expression is imo more clear and direct.

> so, the NP-container is actually mandatory.

I disagree.  One of the sub nodes must exist, and thus the NP
container is needed.  But the container itself is not mandatory.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 22 Apr 2014 09:32:51 -0700

> container is needed.  But the container itself is not mandatory.
Agreed.
NP containers are not intuitive and it takes a little time to understand the
transparency issues.  There is no need to insist that all NP containers
always exist, although that might make them easier to understand.

I think you are objecting to this construct:

   container nonsense {
      mandatory true;
   }

This would be legal but has no semantics.

Actually, an NP container has some semantics.
It represents a collection.  This is real wrt/ naming isolation,
must/when expressions, access control, and data retrieval filtering.

So the statement above could be interpreted as a mandatory collection.
It makes perfect sense for a config=false NP container. It seems to make
sense for config=true as well.

Andy

---

From: Per Hedeland <[email-masked]>
Date: Wed, 23 Apr 2014 08:10:45 +0200

> They do mean different things! The "mandatory" statement is a schema constraint, and sec. 3.1 then explicitly states that "a" is a mandatory node. In contrast, "must" is a semantic constraint that…

Well, in order to not get bogged down by theoretical discussions that
I'm probably not smart enough to understand the relevance of, let me
rephrase as "... radically different things regarding the enforcement of
constraints on valid configuration data, as described in section 8 of
RFC 6020 - in particular regarding any requirement that leaf "b" must
exist in a valid configuration data tree".

This aspect of the meaning of the first construct above is exactly
specified by section 7.6.5 of 6020 (incidentally, without using either
of the concepts "schema constraint" or "mandatory node"). It has no
dependency on whether container "a" exists in the data tree or not, but
it has a dependency on whether container "p" exists in the data tree or
not. I assume we have no disagreement here.

The same aspect of the meaning of the second construct is supposedly
specified by section 7.5.3, which indeed does say

    If a data node does not exist in the data tree, and it does not have
    a default value, its "must" statements are not evaluated.

I.e. it would appear to support "your view". However this interpretation
actually results in "this aspect of the meaning" being undefined by
6020, since it depends on how/when a particular server implementation
creates or deletes np-containers. Thus an <edit-config> that creates "p"
but doesn't set a value for "b" may succeed with one server
implementation (because it didn't create "a") but fail with another
(because it did create "a"). This is obviously not interoperable.

> To support your view, what's essentially needed is to state that NP-containers are always part of the default contents.

I doubt that this is precisely what is needed - at the very least, the
"closest ancestor node in the schema tree that is not a non-presence
container" must come into play, as it does in 7.6.5, 7.7.3, and 7.9.4.

But *something* is definitely needed - and I would suggest that this
issue is separate from the one that started this discussion, the
possibility of having 'mandatory' as substatement to 'container' - and
that this should be reflected in the "issue list".

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Apr 2014 09:39:56 +0200

> Well, in order to not get bogged down by theoretical discussions that

This is no theoretical discussion, this is how off-the-shelf XPath processors work. If you generate DSDL schemas for the second version, Schematron validation will do exactly that: If the validated document contains only <p/>, no error will be reported, while for

<p>
  <a/>
</p>

a validation error will be reported.

In contrast, the "mandatory" statement is mapped straight to RELAX NG, so for the first variant an error will be reported in both cases.

> not. I assume we have no disagreement here.

Right.

> (because it did create "a"). This is obviously not interoperable.

I agree this is not nice and needs to be addressed.

> container" must come into play, as it does in 7.6.5, 7.7.3, and 7.9.4.

Yes, exactly as it does for "normal" defaults.

I think YANG 1.1 spec should also clearly distiguish between the use of the default contents

1. for validation purposes where the XML infoset has to be augmented with default contents, and

2. in NETCONF/RESTCONF messages where the default contents may be omitted (depending on with-defaults setting).

I think the sentence in sec. 7.5.8 about deleting empty NP containers was intended for #2.

Agreed.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 23 Apr 2014 07:01:20 -0700

On Wed, Apr 23, 2014 at 12:39 AM, Ladislav Lhotka <[email-masked]> wrote:

> >> They may not have semantics from YANG's viewpoint but they are still
> nodes in the data tree that either exist or not, and as such influence the
> context for XPath evaluation.
> >>
> >> They do mean different things! The "mandatory" statement is a schema
> constraint, and sec. 3.1 then explicitly states that "a" is a mandatory
> node. In contrast, "must" is a semantic constraint that (as any XPath
> expression) has to be evaluated in the well-defined context of a concrete
> XML instance document. And it is then significant whether the expression's
> context node exists or not.
> >
> This is no theoretical discussion, this is how off-the-shelf XPath
> processors work. If you generate DSDL schemas for the second version,
> Schematron validation will do exactly that: If the validated document
> contains only <p/>, no error will be reported, while for
>
> <p>
>   <a/>
> </p>
>
> In contrast, the "mandatory" statement is mapped straight to RELAX NG, so
> for the first variant an error will be reported in both cases.
>

I do not understand the concerns here.
If you have a must or when expression that tests the existence of
a meaningless container, then that is just garbage-in/garbage-out.
Don't write XPath that depends on NP containers.  Make sure the
tests are for meaningful nodes.

> >
> >> It has been a recurring issue in the previous discussions about "must"
> and especially "when" that the meaning of XPath expressions is being sort
> of shifted to the schema, under the assumption that an implementation can
> (temporarily) insert dummy nodes here and there in the instance document,
> and that the gentle implementor will figure out the intentions of the data
> modeller. I think this is a wrong approach that can lead to
> interoperability problems and race conditions.
> >>
> >> To support your view, what's essentially needed is to state that
> NP-containers are always part of the default contents.
> >
>

Why?
So XPath expressions that test for the existence of meaningless containers
will "work correctly"?

> > that this should be reflected in the "issue list".

The current text says that servers MAY delete empty NP-containers
since they are meaningless and just take up space in the config tree.
Are you saying you want to change that so servers MUST
create NP containers?

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Apr 2014 16:18:45 +0200

> tests are for meaningful nodes.

It is not about testing existence of containers. The problem is with XPath expressions whose context nodes are NP containers: depending on whether such a container exists in the instance data tree, the must constraint may or may not be applied - see my example above.

> will "work correctly”?

No - so that XPath expressions will be applied in a deterministic way.
.
> Are you saying you want to change that so servers MUST

If the NP container has a must statement as its child, then

- if the server deletes the container from the data tree, the must constraint doesn’t apply

- otherwise, the container stays in the data tree and the must constraint applies.

This is hardly acceptable.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 23 Apr 2014 07:27:59 -0700

On Wed, Apr 23, 2014 at 7:18 AM, Ladislav Lhotka <[email-masked]> wrote:

> > In contrast, the "mandatory" statement is mapped straight to RELAX NG,
> so for the first variant an error will be reported in both cases.
> >
> It is not about testing existence of containers. The problem is with XPath
> expressions whose context nodes are NP containers: depending on whether
> such a container exists in the instance data tree, the must constraint may
> or may not be applied - see my example above.
>
> >
> >
> > >
> > >> It has been a recurring issue in the previous discussions about
> "must" and especially "when" that the meaning of XPath expressions is being
> sort of shifted to the schema, under the assumption that an implementation
> can (temporarily) insert dummy nodes here and there in the instance
> document, and that the gentle implementor will figure out the intentions of
> the data modeller. I think this is a wrong approach that can lead to
> > 2. in NETCONF/RESTCONF messages where the default contents may be
> omitted (depending on with-defaults setting).
> >
> - if the server deletes the container from the data tree, the must
> constraint doesn't apply
>
Again -- garbage-in/garbage-out.
YANG allows plenty of constructs, such as top-level mandatory nodes,
which "in theory" are completely broken and unacceptable. So what.

Don't put must-stmt in an empty NP container.
This is an extremely minor corner-case.  In general, the server does
not want meaningless nodes cluttering up the database.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 23 Apr 2014 16:34:10 +0200 (CEST)

> Don't put must-stmt in an empty NP container.

This isn't the issue.  Here's an example of the problem:

  container top {
    presence "...";
    container foo {
      must "a or b";
      leaf a { ... }
      leaf b { ... }
    }
  }

When /top is created, we want a or b to exist.

The behaviour is not clear from RFC 6020.  We need to make this clear.

(I'll add this as a separate clarification issue)

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 23 Apr 2014 07:36:14 -0700

> > > that this should be reflected in the "issue list".

I agree it is an open issue.
I think some implementations actually have the server create NP containers,
not just delete them.  I am more concerned with consistent handling of
the create and delete operations than XPath.  If the server creates an
NP-container on some implementations then a client create operation will
fail,
depending on the implementation. (Unlike with-defaults, there is no
basic-mode
defining what the server will do.)

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 23 Apr 2014 07:39:25 -0700

> When /top is created, we want a or b to exist.
So use the correct XPath:

 container top {
    presence "...";
    must "foo/a or foo/b";
    container foo {
      leaf a { ... }
      leaf b { ... }
    }
  }

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Apr 2014 16:46:02 +0200

> So use the correct XPath:

Well, that’s an option, too. However, nothing in 6020 indicates that Martin’s version is incorrect.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 23 Apr 2014 17:03:23 +0200 (CEST)

> So use the correct XPath:

This is besides the point - the behavior of the construct above should
be clear from the spec, don't you agree?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 23 Apr 2014 08:10:47 -0700

> be clear from the spec, don't you agree?

Sure the text should always be 100% clear.

My point is, the correct way to test for must constraints
if container 'top' exists is to put the must-expr in container 'top'.
The first must-stmt is just plain wrong if you want to test for foo/a or
foo/b,
when the parent node exists.

Andy