Subject: [core] Comments on draft-ietf-core-yang-cbor-00

From: Andy Bierman <[email-masked]>
Date: Mon, 16 May 2016 16:13:19 -0700

Hi,

Here are my comments on the YANG to CBOR draft

Andy

-------------------------------------------

sec. 3: para 4:

   The end user of this mapping specification
   can mandate the use of a specific key type or a specific subset of
   key types.

C1)  How does the end-user do this?

-----------------------------------------

4.2.  The "container" Schema Node

   Collections such as containers, list instances, notifications, RPC
   inputs, RPC outputs, action inputs and action outputs MUST be encoded
   using a CBOR map data item (major type 5).  A map is comprised of
   pairs of data items, with each data item consisting of a key and a
   value.

C2) Add a comment that "key" refers to the child node identifier
and value refers the the value of the child node. It does not
refer to the YANG term "key".

----------------------------------------
sec 4.2:

   *SIDs as keys*

   Keys implemented using SIDs MUST be encoded using a CBOR unsigned
   integer (major type 0)

  2nd bullet:
    Delta values may result in a negative number, clients and servers
      MUST support negative deltas.

C3) Doesn't 2nd bullet require CBOR type 1?

Summary: KEY ENCODING (3 variants):
   SID:     Major type 0 or 1
   string:  Major type 3
   hash:    Major type 2

--------------------------------------------

5.3.  The "decimal64" Type

   Leafs of type decimal64 MUST be encoded using either CBOR unsigned
   integer (major type 0) or CBOR signed integer (major type 1),
   depending on the actual value.  The position of the decimal point is
   defined by the fraction-digits YANG statement and is not available in
   the CBOR encoding.

C4) How are decimal64 values that start with zero encoded?

   leaf foo {
     type decimal64 {
       fraction-digits 3;
     }
     default 0.005;
   }

C5) You probably mean to convert the deciaml64 to a signed
or unsigned integer:

   1.005 == 1005

Then it is up to the decoder to know the YANG schema so it
can adjust based on the fraction-digits value.

---------------------------------------------------

5.6.  The "enumeration" Type

   Leafs of type enumeration MUST be encoded using a CBOR unsigned
   integer data item (major type 0).

C6) Need to say that the implicit or explicit value-stmt value
is used, not the enumeration name like in XML or JSON

C7) Enumeration value is allowed to be negative, so major type 1 also
allowed

------------------------------------------------------

5.7.  The "bits" Type

   Leafs of type bits MUST be encoded using a CBOR byte string data item
   (major type 2).

C8) Need to say that the implicit or explicit position-stmt value
is used, not the bit name like in XML or JSON

------------------------------------------------------

5.8.  The "binary" Type

   Leafs of type binary MUST be encoded using a CBOR byte string data
   item (major type 2).

C9) What string format is used? base64? raw UTF-8?

------------------------------------------------------

5.12.  The "union" Type

   Leafs of type union MUST be encoded using the rules associated with
   one of the types listed.

C10) This does not really work because so many YANG types use
the same CBOR major encoding

   type union {
      type int32;
      type enumeration {
        enum A { value -5; }
        enum B { value 3; }
      }
      type bits {
        bit X { position 1; }
        bit Y { position 3; }
      }
      type decimal64 {
        fraction-digits 2;
      }
   }

How do I know if '3' is for int32, enum B or bit Y?
How do I know 103 is really decimal64 "1.03" and not
an int32 "103"?

C11)  Need to specify that types are evaluated in YANG
statement order (e.g., check int32, then enumeration, then bits)

--------------------------------------------------------

sec. 5.13, para 5

   When SIDs identify a YANG list, the presence of the key(s) for this
   list is optional.  When the key(s) are present, the targeted instance
   within this list is selected.  When the key(s) are absent, the entire
   YANG list is selected.

C12) A YANG instance-identifier MUST identify exactly 1 instance.
It cannot specify all instances of a YANG list.

C13) If instance-identifier used in a union, no way to tell that
SID, name, or hash encoding is really an instance-identifier
and not a string or a number.

C14) How are the keys specified?  The CoMI draft specifies a URI encoding
but nothing in CBOR. Seems like they would have to be a separate string
(keys=A,14,fred).  Examples (e.g. for SID) using an array are not clear.

-----------------------------------------------------------

Attributes:

C15) XML and YANG/JSON allow meta-data to be sent with
the YANG data.  (e.g. NETCONF "operation" or "default" attributes)
How would meta-data be encoded in CBOR?

-----------------------------------------------------------

YANG List Keys:

C16) XML and YANG/JSON require YANG list keys to be encoded first
even if they are not defined first in the list.  This allows server
code to dispatch or do entry lookup without requiring buffering.
Should CBOR encoding of YANG lists require the same?

---

From: Michel Veillette <[email-masked]>
Date: Tue, 17 May 2016 18:00:58 +0000

Hi Andy and thanks for your comments.
See [MV] bellow for my answers.

I'll wait for your reply before applying any changes to the latest draft.
http://core-wg.github.io/yang-cbor/draft-ietf-core-yang-cbor-latest.html

Regards,
Michel

From: core [mailto:[email-masked]] On Behalf Of Andy Bierman
Sent: May-16-16 7:13 PM
To: Core <[email-masked]>
Subject: [core] Comments on draft-ietf-core-yang-cbor-00

Hi,

Andy

sec. 3: para 4:

[MV]
The "end user" in this context is typically a specification.
For example, draft-veillette-core-cool-01 mandate SID as key type.

We should probably use something else than "end user"
Can you propose a new text to clarify the intent of this sentence?

[MV] I proposing to add the following text. Fell free to propose an alternate text.

"Each key within the CBOR map is set to a data node identifier, each
value is set to the value of this data node instance."

   *SIDs as keys*

  2nd bullet:

[MV] This issue is already fixed in the latest draft, see
http://core-wg.github.io/yang-cbor/draft-ietf-core-yang-cbor-latest.html#rfc.section.4.2

   leaf foo {
     type decimal64 {
       fraction-digits 3;
     }
     default 0.005;
   }

[MV] If I use the default value as example, the value 0.005 is encoded as 0x05 in CBOR.
If a textual representation is required, the logic performing this conversion needs to add the leading zero(s) and the decimal point.

   1.005 == 1005

[MV] Correct.

[MV]
Correct for both comments.
The first sentence need to be fixed and a clarification need to be added about the automatic assignment algorithm. I propose:

"Leafs of type enumeration MUST be encoded using a CBOR unsigned integer (major type 0) or CBOR signed integer (major type 1), depending on the actual value. Enumeration values are either explicitly assigned using the YANG statement "value" or automatically assigned based on the algorithm defined in [RFC6020biss] section 9.6.4.2.

5.7.  The "bits" Type

[MV]
Correct, I propose:

"Leafs of type bits MUST be encoded using a CBOR byte string data item (major type 2). The position of each bit is either explicitly assigned using the YANG statement "position" or automatically assigned based on the algorithm defined in [RFC6020biss] section 9.7.4.2."

5.8.  The "binary" Type

[MV] No special format, just binary, see example.

5.12.  The "union" Type

   type union {
      type int32;
      type enumeration {
        enum A { value -5; }
        enum B { value 3; }
      }
      type bits {
        bit X { position 1; }
        bit Y { position 3; }
      }
      type decimal64 {
        fraction-digits 2;
      }
   }

[MV] Ouch, very good point.
I'm proposing to use CBOR tags for the following list of datatypes to avoid this kind of ambiguities.

·        bits

·        decimal64

·        enumeration

·        identityref

·        instance-identifier

·        leafref  (Only if if the datatype of the referenced leaf require a CBOR tag)

For the following list of  datatypes, they can be used directly without CBOR tag.

·        binary

·        boolean

·        empty

·        int8

·        int16

·        int32

·        int64

·        string

·        uint8

·        uint16

·        uint32

·        uint64

-------
Propose text:

Leafs of type union MUST be encoded using the rules associated with one of the types listed.
When use in a union, the following list of YANG datatypes are prefixed by CBOR tag to avoid confusion
between different YANG datatypes encoded using the same CBOR major type.

o bits
o decimal64
o enumeration
o identityref
o instance-identifier
o leafref  (Only if the datatype of the leaf referenced using the "path" YANG statement require a CBOR tag)

See section 7.1 for more information about CBOR tags.

-------
7.  IANA Considerations

7.1.  Tags Registry

This specification require the assignment of CBOR tags for the following YANG datatypes.
These tags are added to the Tags Registry as defined in section 7.2 of RFC 7049.

| Data Item | Semantics | Reference
| bits | YANG bits datatype | RFC6020biss, draft-ietf-core-yang-cbor
| decimal64 | YANG decimal64 datatype | RFC6020biss, draft-ietf-core-yang-cbor
| enumeration | YANG enumeration datatype | RFC6020biss, draft-ietf-core-yang-cbor
|  identityref | YANG identityref datatype | RFC6020biss, draft-ietf-core-yang-cbor
| instance-identifier | YANG instance-identifier datatype | RFC6020biss, draft-ietf-core-yang-cbor
|  leafref  | YANG leafref  datatype | RFC6020biss, draft-ietf-core-yang-cbor

[MV] I don’t understand this comment or proposed solution.

sec. 5.13, para 5

[MV] The definition of a list in RFC 6020 seem effectively to back-up this conclusion.

"list: An interior data node that may exist in multiple instances
in the data tree. A list has no value, but rather a set of child nodes."

Are you aware of any other places where this limitation is described?

If this is the case, the following sentences need to be removed and re-introduced as extension within
draft-veillette-core-cool when instance-identifiers are used in the context of the FETCH or iPATCH methods.

"When SIDs identify a YANG list, the presence of the key(s) for this list is optional.

When the key(s) are present, the targeted instance within this list is selected.

When the key(s) are absent, the entire YANG list is selected."

Note:
draft-veillette-core-cool already add two extensions to instance-identifier in the context of a FETCH or iPATCH,
we will need to add a third one. I still believe that using instance-identifier as the base construct
for FETCH and iPATCH make sense in order to minimize implementation footprint even if extensions are needed.

[MV] Addressed by my previous answer.

[MV] When key(s) are needed to qualify the instance-identifier, a CBOR array is used instead of a simple CBOR unsigned integer.
The first entry within the CBOR array is the SID, the next entries are the key(s) encoded using the rules defined for their datatypes.
There is no text conversion require to minimise the implementation footprint.

The following example:
"/ietf-system:system/authentication/user[name='bob']/authorized-key[name='admin']/key-data"

Is encoded as follow in CBOR:
[1721, "bob", "admin"]

Can you propose extra text which will clarify this item?

Attributes:

[MV] Can you provide me some reference to this topic.
I didn't find any references to "operation" and "default" in "draft-ietf-netmod-yang-json"

YANG List Keys:

[MV] This is not possible for both JSON and CBOR.
The ordering of items within JSON objects or CBOR maps are arbitrary.

This is addressed in "draft-ietf-netmod-yang-json" section 5.4

"Unlike the XML encoding, where list keys are required to precede any
other siblings within a list entry, and appear in the order specified
by the data model, the order of members in a JSON-encoded list entry
is arbitrary because JSON objects are fundamentally unordered
collections of members. "

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 May 2016 13:16:20 -0700

Hi Michel,

All of your proposed changes look good.
Re (C11): YANG 1.1 already specifies how the union type must be evaluated.
(In the order the member types are listed).
However, the results may not be the same for all encodings.

E.g,

   type union {
      type string;
      type int32;
   }

In XML, the element <foo>42</foo> would match the first type 'string'.
In JSON or CBOR, the encoding will specify string vs. integer, so
the positive integer 42 would not match type 'string'.

  { "foo": 42 }
  { "foo": "42" }

This is a problem for internal instrumentation that wants to be
encoding-independent,
but I do not know what to do about it.  We could force a type conversion
like XPath,
but this seems broken.

You should always design a YANG union so the pickiest types are listed
first,
always make string last.

On Tue, May 17, 2016 at 11:00 AM, Michel Veillette <
[email-masked]> wrote:

> *From:* core [mailto:[email-masked]] *On Behalf Of *Andy Bierman
> *Sent:* May-16-16 7:13 PM
> *To:* Core <[email-masked]>
> *Subject:* [core] Comments on draft-ietf-core-yang-cbor-00
>
>
>
> Hi,
>
>
>

Andy

> If a textual representation is required, the logic performing this
> conversion needs to add the leading zero(s) and the decimal point.
>
>
>
> The first sentence need to be fixed and a clarification need to be added
> about the automatic assignment algorithm. I propose:
>
>
>
> "Leafs of type enumeration MUST be encoded using a CBOR unsigned integer
> (major type 0) or CBOR signed integer (major type 1), depending on the
> actual value. Enumeration values are either explicitly assigned using the
> YANG statement "value" or automatically assigned based on the algorithm
> defined in [RFC6020biss] section 9.6.4.2.
>
>
>
> "Leafs of type bits MUST be encoded using a CBOR byte string data item
> (major type 2). The position of each bit is either explicitly assigned
> using the YANG statement "position" or automatically assigned based on the
> algorithm defined in [RFC6020biss] section 9.7.4.2."
>
>
>
> draft-veillette-core-cool already add two extensions to
> instance-identifier in the context of a FETCH or iPATCH,
>
> we will need to add a third one. I still believe that using
> instance-identifier as the base construct
>
> [MV] When key(s) are needed to qualify the instance-identifier, a CBOR
> array is used instead of a simple CBOR unsigned integer.
>
> The first entry within the CBOR array is the SID, the next entries are the
> key(s) encoded using the rules defined for their datatypes.
>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 May 2016 23:32:28 +0200

> but this seems broken.

Ouch.

I would have said the breakage is clearly on the XML side, except that
we have the same problem between, say, integer and decimal64.

I think the breakage we really need to avoid is where XML can tell the
types apart and the CBOR mapping cannot.  Is there such a case?

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 May 2016 14:41:56 -0700

> types apart and the CBOR mapping cannot.  Is there such a case?
No, I think the problems are with XML.
There are a few derived types like "xpath1.0" in ietf-yang-types.yang
that rely on prefix to namespace mappings that cannot be represented in
CBOR,
but I think the base types are OK.

Grüße, Carsten

Andy

---

From: Michel Veillette <[email-masked]>
Date: Wed, 18 May 2016 03:19:29 +0000

Hi Andy

The draft have been updated to address your comments.

Diff available at:
https://github.com/core-wg/yang-cbor/commit/c50b6840f4c87a0041e4a41cf05a09278c99a5a8
Latest document available at:
http://core-wg.github.io/yang-cbor/draft-ietf-core-yang-cbor-latest.html

Thanks again for reviewing this draft.

Michel

From: Andy Bierman [mailto:[email-masked]]
Sent: May-17-16 5:42 PM
To: Carsten Bormann <[email-masked]>
Cc: Michel Veillette <[email-masked]>; Core <[email-masked]>
Subject: Re: [core] Comments on draft-ietf-core-yang-cbor-00

On Tue, May 17, 2016 at 2:32 PM, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:

Ouch.

No, I think the problems are with XML.
There are a few derived types like "xpath1.0" in ietf-yang-types.yang
that rely on prefix to namespace mappings that cannot be represented in CBOR,
but I think the base types are OK.

Grüße, Carsten

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 21 May 2016 09:44:55 +0200

> the same CBOR major encoding

The receiving side will resolve the union type by the first union member
type that matches the CBOR-encoded value. It's the same as in JSON encoding.

> How do I know if '3' is for int32, enum B or bit Y?

If it's major type 0 or 1, it will be int32 in this case because int32 is the
first applicable member type.

> How do I know 103 is really decimal64 "1.03" and not

If it's major type 0 or 1, then it will be again int32. However, 2147483648
would become 21474836.48 because that integer doesn't fit the int32 range.

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 21 May 2016 10:30:58 +0200

> type that matches the CBOR-encoded value. It's the same as in JSON encoding.

Or, at least, similar -- we are not always making the same serialization
decisions.

The weird aspect of this part of YANG is that the actual validity (and
specific semantics and thus usefulness) of a union in the YANG module
depends on the specifics of the serialization below.
More frankly: Tying the semantics of a modeling language to one specific
serialization mechanism is an utterly broken design.
I'm not sure that it can be fixed, though, so we'll likely have to work
around this breakage.

What we were trying to figure out was the extent of the situations where
a YANG module specifier might rely on the XML-specific way of resolving
unions.

Examples that seem painful:

1)

 type union {
      type enumeration {
         enum one; /* value 0 */
         enum two; /* value 1 */
      }
      type uint32;
 }

Here, the XML will have the string "one" or "two", or a decimal string,
which is unambiguous.  The JSON encoding also has the string for
enumerations, which we obviously want to replace by the numeric value in
a compact encoding.

We were thinking about possibly adding CBOR tags to disambiguate this.
This becomes more insidious if this is an evolution from an earlier
version that just said:

 type enumeration {
   ...
 }

(We wouldn't want to sprinkle in tags just in case a type might possibly
evolve into a union later.)

2)

Also, just having a tag saying "this is an enumeration value" does not
work for this case:

 type union {
      type enumeration {
         enum one; /* value 0 */
         enum two; /* value 1 */
      }
      type enumeration {
         enum alpha; /* value 0 */
         enum beta; /* value 1 */
      }
 }

We haven't found a way yet to tag this case that is robust against
further evolution of the YANG module.

Clearly, YANG module specifiers will not rely on this when they are
cognizant of concise encodings.  However, relying on specification
writers not doing this would mean that YANG modules have to be vetted
for mistakes of this kind before they can be used in a CBOR environment
after an exclusively XML/JSON life.  (Maybe that actually *is* the
answer -- if we could get pyang to detect this case and issue a warning,
that might be enough.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 21 May 2016 10:36:18 +0200

> type that matches the CBOR-encoded value. It's the same as in JSON encoding.

Oh, and I also have a problem with requiring YANG-CBOR decoders to do
this kind of "matching", which is:

-- complex (you need a full constraint evaluator)
-- not necessarily well-defined (prove that it is!)
-- not necessarily even possible at the time of decoding (!)

The example in Section 9.12.4 of draft-ietf-netmod-rfc6020bis-12.txt of
a union data item retroactively (i.e., after decoding!?) turning into a
string when the leaf that it was originally referencing was deleted is
hilarious.

Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 21 May 2016 13:28:57 +0200

> depends on the specifics of the serialization below.

YANG's native serialization of values are untyped strings.

> serialization mechanism is an utterly broken design.

Not necessarily. It may start to look broken once you have
serializations that are typed and for some reasons people think the
type information should impact how YANG works (that assumes values
serialized as untyped strings). Note that type rules differ for
different serializations so you actually have no chance to get away
without any weird corner cases.

> I'm not sure that it can be fixed, though, so we'll likely have to work

I disagree that YANG is broken.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 21 May 2016 15:08:38 +0200

Hi Jürgen,

"broken" is indeed a strong word -- I deliberately used it as a way to
provoke reactions, and that seems to have worked :-).

> YANG's native serialization of values are untyped strings.

Hmm.  YANG is increasingly being positioned as a data modeling language.
Unless my entire world revolves around TCL :-), my data models need a
richer set of types than just strings.

Having serialization details invade the (information and data) modeling
levels is exactly what brought down XML, and finding that YANG inherits
(a mild form of) these problems is very worrisome.

> without any weird corner cases.

Separation of concern between the data model level and the serialization
level is an important aspect of a data modeling language -- processing
rules should operate on the data model, not on the serialization (again,
the related XML problem is rearing its head).

> I disagree that YANG is broken.

Well, I take that word back, but I do think this is one place where YANG
as a data modeling language has a serious shortcoming, and it does not
look likely that the YANG-CBOR mapping can work around that shortcoming
in a particularly clean way.

As a way of coping, we probably should develop some "best practices"
(read: describe problematic cases) around the use of YANG unions that
then at least can be considered by the YANG doctors for IETF modules.

Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 21 May 2016 16:23:23 +0200

> richer set of types than just strings.

YANG has a rich set of types. The native serialization is a textual
format.
 
> the related XML problem is rearing its head).

I claim YANG itself is fine; the JSON encoding (the first encoding
that has some (and please also not incompatible) typed encoding ended
up needing YANG type information to do things right; this was not my
favourite choice but I lost this debate.
 
> in a particularly clean way.

CBOR either does that same as JSON (which I agree leeds to some ugly
cases) or CBOR hands at the end a string value over to the YANG type
system (in particular in cases where it matters to get a string
value).

> then at least can be considered by the YANG doctors for IETF modules.

If this can't be avoided, it is better to rely on tools rather than on
YANG doctors.

/js

---

From: Andy Bierman <[email-masked]>
Date: Sat, 21 May 2016 08:29:33 -0700

> a YANG module specifier might rely on the XML-specific way of resolving

I agree YANG is broken is the sense that the union type was not
designed to work if the protocol encoding is not XML.
Since every value node in XML is a string, the sender and
receiver cannot get out of sync wrt/ which member type matches.

This is a legal union type even though in XML type 'int32' would never be
used
because 'string' would match everything.

 leaf foo {
   type union {
      type string;
      type int32;
   }
 }

This is a problem for config nodes:
   JSON/CBOR client sets { "foo" : 42 }
   XML client reads back <foo>42</foo>

JSON (and CBOR) will send { "foo":42 }, not { "foo":"42" } if they intend to
send the number, but XML cannot do the same.  There is no problem
unless XML is being used in addition to JSON or CBOR.

Some solution choices:

  1 - do nothing and let XML and JSON/CBOR be broken together
     (solution is do not use XML and JSON/CBOR together in the same server)
  2 - force receiver to convert JSON/CBOR to a string for union type
matching
  3 - tag the data somehow so the receiver knows which union member type is
being sent
  4 - write YANG guidelines on how to design union types that will produce
the same
     match with all encoding schemes

IMO CBOR has to solve the enumeration and bits issues with (3) but
for the problem in general, just do (4)

 Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 May 2016 13:07:15 +0200

> Or, at least, similar -- we are not always making the same serialization

I meant that the rule in section 9.12 of rfc6020bis still applies:

   a value representing a union data type is validated consecutively
   against each member type, in the order they are specified in the
   "type" statement, until a match is found.

However, deciding about validity of a value involves not only YANG type
(or data model info in general) but also rules of the given
serialization format.

I don't think it's a big practical problem though. The sending side
should know the "real" type of a value being and the serialization
format, so it should be able to do the right thing. 

> depends on the specifics of the serialization below.

This is not specific to unions: e.g., if a YANG leaf "foo" has a numeric
type, then a JSON value

"foo": "42"

will be rejected, and so will be a CBOR value with major type 3.

> enumerations, which we obviously want to replace by the numeric value in

Right, so the problem is not only due to "everything is string in XML".

> We were thinking about possibly adding CBOR tags to disambiguate this.

This might be a good idea anyway because resolving a union type may
involve, e.g., (repeated) regular expression matching.

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 23 May 2016 13:14:08 +0200

> involve, e.g., (repeated) regular expression matching.

Right, minimizing the complexity of the matcher is a good objective.

Now, how do we use tags to identify the alternatives in the union?
As far as I understand, these can be moved around when the module is
updated.
How do people handle this in their software?

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 May 2016 13:22:59 +0200

> YANG's native serialization of values are untyped strings.

To my knowledge, this hasn't been stated as a design feature for YANG,
it's just a side effect of XML being the only serialization that was
considered when YANG was being designed.

> I disagree that YANG is broken.

+1, although I think it would be useful to have a general mechanism for
signaling the actual type of a value - one option is to use a metadata
annotation.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 May 2016 13:27:19 +0200

> -- not necessarily even possible at the time of decoding (!)

Right, tagging the values is a better option.

> string when the leaf that it was originally referencing was deleted is

Agreed.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 23 May 2016 14:06:27 +0200

> On 23 May 2016, at 13:14, Carsten Bormann <[email-masked]> wrote:
> 
> Ladislav Lhotka wrote:

I think they mostly follow the procedure as specified in 6020. Unions are usually not defined to make this really hard. For unions like "inet:host" or "inet:ip-address" it often needn't be done at all because many library functions directly accept all member types.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 May 2016 14:36:13 +0200

> considered when YANG was being designed.

We use string representations of values in other contexts, e.g., when
writing pattern restrictions or when evaluating xpath expressions. We
also do define canonical string formats for our types.

Note, this does not mean an implementation has store internally all
values as strings but when it matters, the string representation of
values defines the behavior. Except for unions in JSON...

My take has been that non-XML encodings may take advantage of being
able to present data mode 'efficiently' (CBOR) or to make it easier
for applications / libraries (JSON). For me, encodings that change how
we interpret types at the YANG level are somewhat surprising.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 May 2016 14:40:39 +0200

>      match with all encoding schemes

I think 2 is the right short-term approach. It does not mean an
implementation has always to convert to a string but an implementation
needs to behave as if it has converted to a string.

> for the problem in general, just do (4)

Option 3 may be a long-term solution but this is non-trivial to work
out and to introduce with a backwards compability perspective in mind.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 24 May 2016 16:25:13 +0200

> On 24 May 2016, at 14:36, Juergen Schoenwaelder <[email-masked]> wrote:
> 

Actually, pattern restrictions are only possible for the "string" type.

And XPath is not limited to strings, we can write, e.g.,

    must ". mod 2 = 0";

> we interpret types at the YANG level are somewhat surprising.

The problem is that XML, JSON and CBOR are not just different encodings (as in Unicode) but rather different ways for representing structured data, and each has different priorities and semantics. I don't think that "gleichgeschaltetes" JSON or CBOR, where everything is a string, is an attractive option.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 May 2016 16:29:24 +0200

> And XPath is not limited to strings, we can write, e.g.,

But the expression is at least conceptually evaluated on a string
representation of what is being interpreted as a number.
 
> The problem is that XML, JSON and CBOR are not just different encodings (as in Unicode) but rather different ways for representing structured data, and each has different priorities and semantics.…

And I did not suggest that. All I said is that encodings should not
change how the YANG type system works. I continue to find that broken.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 24 May 2016 17:08:58 +0200

> On 24 May 2016, at 16:29, Juergen Schoenwaelder <[email-masked]> wrote:
> 

Hmm, I would say it's just the other way around: for example, XPath expression "foo = 0" evaluates to true even if "foo" happens to have YANG "string" type and value of "00".

> change how the YANG type system works. I continue to find that broken.

I don't think CBOR or JSON really change how YANG type system works. They just provide some extra information that (unfortunately) isn't available in XML serialization. Rather than blaming non-XML serializations, we should think about a generic way for signalling the actual member type that can be used whenever the chosen serialization falls short. If we have, e.g.

leaf foo {
  type union {
    type string;
    type uint8;
  }
}

then it should IMO be possible to pass a number, no matter what the encoding is. So nothing extra needs to be done in JSON and CBOR, but in XML encoding the data could be sent as

<foo member-type="2">42</foo>

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 May 2016 17:55:33 +0200

On 24 May 2016 at 17:09:01, Ladislav Lhotka ([email-masked]) wrote:
<foo member-type="2">42</foo> 
That brings up a question:

Are there any rules about the evolution of unions?

Can a YANG implementation rely on the relative position of union members staying the same over the evolution of modules?

Can a YANG implementation rely on a specific position in a union for what used to be the only permissible type in a previous version of a module?

Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 24 May 2016 18:04:33 +0200

> Can a YANG implementation rely on a specific position in a union for what used to be the only permissible type in a previous version of a module?

I did not find specific text (neither in the YANG 1.1 specification
nor in the guidelines document). I do not know whether tools discover
and flag reorderings. Well, some reorderings may be OK, others may not
be OK. Perhaps the simplest thing would be to put text into the
guidelines document in any case. Avoiding union members with
overlapping value sets may generally be a good idea.

/js

---

From: Michel Veillette <[email-masked]>
Date: Tue, 24 May 2016 17:02:26 +0000

Hi Andy

I propose that we articulate a solution around your proposed solution #3 and #4.

If we implements YANG built-in type tagging (Solution #3), we can remove any ambiguities between the different YANG built-in types introduced by the CBOR encoding.
YANG built-in type tagging is preferred over union member tagging to increase backward compatibility between YANG module revisions.
I propose that the YANG built-in types listed bellow with a * are tagged.

  binary              | major type 2
* bits                | major type 2
  boolean             | major type 7, additional information 20 and 21
* decimal64           | major type 0 and 1
  empty               | major type 7, additional information 22
* enumeration         | major type 0
* identityref         | major type 0 and 1
* instance-identifier | major type 0, 2, 3 or 4
  int8                | major type 0 and 1
  int16               | major type 0 and 1
  int32               | major type 0 and 1
  int64               | major type 0 and 1
  string              | major type 3
  uint8               | major type 0
  uint16              | major type 0
  uint32              | major type 0
  uint64              | major type 0

This proposed CBOR tagging remove any ambiguities between:

·        integer, decimal64, enumerator, identityref and instance-identifier

·        binary, bits and instance-identifier

·        string and instance-identifier

Solution #4 (guidelines) are still needed to address the following issues:

·        evolution of a simple type to a union

o   Not recommended for tagged built-in types (bits, decimal64, enumeration, …) to avoid non backward compatible encoding of older YANG module versions.

·        limitations imposed by the use of multiple union members based on the same built-in type

o   string patterns must be unique

o   integer ranges must be unique

o   enumeration names and values must be unique

o   bits names and positions must be unique

o   multiple instance-identifier are not allowed since uniqueness are not guaranty

Regards,
Michel

Sent: May-21-16 11:30 AM
Cc: Core <[email-masked]>

On Sat, May 21, 2016 at 1:30 AM, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:

This is a legal union type even though in XML type 'int32' would never be used
because 'string' would match everything.

 leaf foo {
   type union {
      type string;
      type int32;
   }
 }

Some solution choices:

  1 - do nothing and let XML and JSON/CBOR be broken together
     (solution is do not use XML and JSON/CBOR together in the same server)
  2 - force receiver to convert JSON/CBOR to a string for union type matching
  3 - tag the data somehow so the receiver knows which union member type is being sent
  4 - write YANG guidelines on how to design union types that will produce the same
     match with all encoding schemes

 Andy

1)

 type union {
      type enumeration {
         enum one; /* value 0 */
         enum two; /* value 1 */
      }
      type uint32;
 }

 type enumeration {
   ...
 }

2)

 type union {
      type enumeration {
         enum one; /* value 0 */
         enum two; /* value 1 */
      }
      type enumeration {

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 May 2016 20:28:56 +0200

Hi Michel,

type tagging does not completely solve the problem:

 type union {
      type enumeration {
         enum one; /* value 0 */
         enum two; /* value 1 */
      }
      type enumeration {

(Both arms of the union would get an enumeration tag, so I still don't
know which arm is intended.)

Also, type tagging carries a serialization cost (small, but
recognizable) for something that really should be solved at the data
model level.  More fundamentally, it means that users of the feature
enumeration etc. are paying a cost for a corner case of the feature
union.  That is almost always the wrong thing to do.

Looking at RFC 6021, there are three unions in there:

   typedef ip-address {
     type union {
       type inet:ipv4-address;
       type inet:ipv6-address;
     }
   }

   typedef ip-prefix {
     type union {
       type inet:ipv4-prefix;
       type inet:ipv6-prefix;
     }
   }

and

   typedef host {
     type union {
       type inet:ip-address;
       type inet:domain-name;
     }
   }

(Yes, that is a union in a union.)

All of these are strings, with varying regexes ("patterns") restricting
the values.  So there is an expectation that receivers of data will do
pattern matching to resolve the union.  (Note that IPv4 addresses match
the regex for domain-name, so matching in order is relevant here.)

I don't think we want to do this for constrained environments.  I also
don't think we want to ship strings around for what really are 4-byte or
16-byte addresses.  Contrast the GRASP draft:

   locator-option /= ipv4-locator-option
   ipv4-locator-option = bytes .size 4
   ; this is simpler than [O_IPv4_LOCATOR, bytes .size 4]

   locator-option /= ipv6-locator-option
   ipv6-locator-option = bytes .size 16

   locator-option /= fqdn-locator-option
   fqdn-locator-option = [O_FQDN_LOCATOR, text]

   locator-option /= uri-locator-option
   uri-locator-option = [O_URI_LOCATOR, text]

(That is not solving exactly the same problem, but is closer to what one
would want in a constrained environment.)  I have no idea how to get
there (GRASP) from here (YANG), but wouldn't it be nice if we had a way
to do this.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 May 2016 21:03:18 +0200

Here is a nice example of a union that is actually a union in the
set-theoretic sense:

  typedef security-model {
    type union {
      type enumeration {
        enum v1  { value 1; }
        enum v2c { value 2; }
        enum usm { value 3; }
        enum tsm { value 4; }
      }
      type int32 {
        range "1..2147483647";
      }
    }
    reference
      "RFC 3411: An Architecture for Describing Simple Network
         Management Protocol (SNMP) Management Frameworks";
  }

(ietf-snmp-common.yang from https://github.com/YangModels/yang)

Note that there is no intention to actually allow a value of 1 to 4 for
the int32 arm.

This is an example of a general pattern that we see in data modeling
languages where the model both tries to define the generic set of values
(positive int32 here) and specific allocations within that set.
(CDDL has .within for this purpose.)

Here it would not be useful to tag the enumeration specially, as all
positive int32 values are intended to be allowed; it is just that four
of those values have a meaning defined in this specification.  If that
specification evolves, we might add more such values in the enumeration,
changing the tag for those with respect to implementations that
previously already have used that value without the specification having
specifically called it out.  Ouch.

(There are 219 occurrences of "type union" in that repo; most of these
are just DRY violations though.)

Grüße, Carsten

---

From: Michel Veillette <[email-masked]>
Date: Tue, 24 May 2016 19:17:31 +0000

Hi Carsten

About the case of a union with multiple enumerations defining overlapped values, (example provided in your email)
I'll argue that the rule consisting of evaluating union members in order until a match is found apply.
The YANG guideline should describe this case and associated limitations, tools like pyang should generate a warning.
It's important to note that this problem of overlaps is not limited to enumerations,
your email mention domain-name which overlap IPv4 address and are both defined as string.

The only instance of multiple enumerations I found so far is "security-model-or-any", however there is no duplicated values.
http://www.netconfcentral.org/modules/ietf-snmp-common/2014-05-06#security-model-or-any.121

I want to note again that YANG "choice / case" are equivalent to union but each arms are explicitly identified (e.g. using a specific SID).
This is an option available to YANG module developers to avoid ambiguities.
The YANG guideline should recommend this approach when appropriate.

About the encoding of IP addresses using a binary field, the only solution I see is to add this option in the corresponding unions in a future revision.
For new modules targeting specifically constrained environments, IP addresses can be defined directly as binary fields.

The YANG module "ietf-cool.yang" already include at least one typedef targeting constrained environments.

  typedef utc-time {
    type uint32;
    description
      "Unsigned 32-bit value representing the number of seconds
      since 0 hours, 0 minutes, 0 seconds, on the 1st of January,
      2000 UTC (Universal Coordinated Time).";
  }

This module can be extended with more typedefs targeting constrained environments.

Regards,
Michel

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: May-24-16 2:29 PM
To: Michel Veillette <[email-masked]>
Cc: Andy Bierman <[email-masked]>; Core <[email-masked]>
Subject: Re: [core] Comments on draft-ietf-core-yang-cbor-00

Hi Michel,

 type union {
      type enumeration {
         enum one; /* value 0 */
         enum two; /* value 1 */
      }
      type enumeration {

(Both arms of the union would get an enumeration tag, so I still don't know which arm is intended.)

Also, type tagging carries a serialization cost (small, but
recognizable) for something that really should be solved at the data model level.  More fundamentally, it means that users of the feature enumeration etc. are paying a cost for a corner case of the feature union.  That is almost always the wrong thing to do.

   typedef ip-address {
     type union {
       type inet:ipv4-address;
       type inet:ipv6-address;
     }
   }

   typedef ip-prefix {
     type union {
       type inet:ipv4-prefix;
       type inet:ipv6-prefix;
     }
   }

and

   typedef host {
     type union {
       type inet:ip-address;
       type inet:domain-name;
     }
   }

All of these are strings, with varying regexes ("patterns") restricting the values.  So there is an expectation that receivers of data will do pattern matching to resolve the union.  (Note that IPv4 addresses match the regex for domain-name, so matching in order is relevant here.)

I don't think we want to do this for constrained environments.  I also don't think we want to ship strings around for what really are 4-byte or 16-byte addresses.  Contrast the GRASP draft:

(That is not solving exactly the same problem, but is closer to what one would want in a constrained environment.)  I have no idea how to get there (GRASP) from here (YANG), but wouldn't it be nice if we had a way to do this.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 May 2016 21:35:02 +0200

Oh, and then there are gems such as:

  typedef ip-filter-num {
    type uint32 {
      range "1..199";
    }
  }

  typedef ip-filter-name {
    type string {
      length "1..63";
    }
  }

  typedef ip-filter-num-and-name {
    type union {
      type ip-filter-num;
      type ip-filter-name;
    }
    description
      "This data provides option of both ACL Name and number";
  }

(brocade-bgp.yang)

I don't think we can help this specification:

  typedef fc-node-name-type {
    type union {
      type common-def:wwn-type;
      type string;
      type common-def:rbridge-id-type;
    }
  }

(with:

  typedef rbridge-id-type {
    type uint32 {
      range "1..239" {
        error-message "Invalid rbridgeId (must be in the range 1-239).";
      }
    }
  }

which would never match according to RFC 6020 rules.)

... or this:

  typedef Char-num {
    type union {
      type string {
        pattern "(\p{IsBasicLatin}|\p{IsLatin-1Supplement})*";
      }
      type uint8;
    }
    description "Takes a character or its ASCII decimal equivalent
                 (0-255).";
  }

I found exactly one example where we'd need a tag (really, this is just
a null-style special value):

    type union {
      type uint32;
      type enumeration {
        enum unbounded;
      }
    }

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 May 2016 07:36:12 +0200

> On 24 May 2016, at 17:55, Carsten Bormann <[email-masked]> wrote:
> 

Sec. 11 in 6020bis says that a type cannot be updated if it changes its semantics, which seems to be the case here.

> Can a YANG implementation rely on a specific position in a union for what used to be the only permissible type in a previous version of a module?

Alternatively, the annotation could specify the member type by name, e.g.

   <host member-type="ietf-inet-types:ipv6-address">...</host>

and in JSON

"host": "..."
"@host": { "member-type": "ietf-inet-types:ipv6-address" }

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 25 May 2016 10:25:38 +0200

> Alternatively, the annotation could specify the member type by name, e.g.

That's great (and we could use SIDs to reduce the bulk) if the type has
a name.  Many of the examples I cited yesterday define the enumerations
in place in the union, without giving them a name.  Apart from
goedelizing these (which *could* be done by the SID process as well),
I'm out of ideas.

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 May 2016 13:18:45 +0200

> On 25 May 2016, at 10:25, Carsten Bormann <[email-masked]> wrote:
> 
> Ladislav Lhotka wrote:

Then we'd have to stick to the numbers. Any reordering of member types would affect the current procedure for union validation, too.

Lada