Subject: [netmod] Fwd: New Version Notification for draft-lhotka-netmod-yang-json-01.txt

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 2 Apr 2013 19:21:49 +0200

Hi,

this revision corrects several mistakes and adds an example for union type.

Lada

Begin forwarded message:

> From: [email-masked]
> Subject: New Version Notification for draft-lhotka-netmod-yang-json-01.txt
> Date: April 2, 2013 7:19:11 PM GMT+02:00
> To: [email-masked]
> 
> 
> A new version of I-D, draft-lhotka-netmod-yang-json-01.txt
> has been successfully submitted by Ladislav Lhotka and posted to the
> IETF repository.
> 
> Filename:	 draft-lhotka-netmod-yang-json
> Revision:	 01
> Title:		 Modeling JSON Text with YANG
> Creation date:	 2013-04-02
> Group:		 Individual Submission
> Number of pages: 14
> URL:             http://www.ietf.org/internet-drafts/draft-lhotka-netmod-yang-json-01.txt
> Status:          http://datatracker.ietf.org/doc/draft-lhotka-netmod-yang-json
> Htmlized:        http://tools.ietf.org/html/draft-lhotka-netmod-yang-json-01
> Diff:            http://www.ietf.org/rfcdiff?url2=draft-lhotka-netmod-yang-json-01
> 
> Abstract:
>   This document defines rules for mapping data models expressed in YANG
>   to configuration and operational state data encoded as JSON text.  It
>   does so by specifying a procedure for translating the subset of YANG-
>   compatible XML documents to JSON text, and vice versa.
> 
> 
> 
> 
> The IETF Secretariat
> 

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Tue, 2 Apr 2013 15:29:51 -0400

>this revision corrects several mistakes and adds an example for union type.

Looks good.  Three questions:

Have you considered including an encoding for the attributes 
that YANG defines (insert/before/after/operation/etc.)?  I
don't see them mentioned in the draft at all.

HAve you considered including a means of expressing namespace
mappings?  It's not clear if you are putting this mapping entirely
outside of the draft, but it seems like a facility worth having.

Can you add an example where one YANG modules augments another?
I can't tell from your text what would happen.  Actually, I can
find nothing on augmentation in the draft.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 2 Apr 2013 22:19:54 +0200

> don't see them mentioned in the draft at all.

I decided to stay away from attributes, for two reasons:

1. I don't know how to encode the binding between such an attribute and the data node (especially leaf) to which it belongs, because in JSON they would have to be siblings, and there might be other sibling leaves.

2. APIs that are based on HTTP methods, as well as e.g. JSON Patch, don't really need these attributes.

> outside of the draft, but it seems like a facility worth having.

I think section 3.1 defines this mapping and the encoding of an explicit namespace id:

<module name>:<local name>

> find nothing on augmentation in the draft.

The mapping is between instance documents, so augments are not special, except that they
may lead to local name collisions where explicit namespace ids are required. But if the local name is unambiguous, the <module>: prefix is not required, even for augmented nodes.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 5 Apr 2013 09:02:58 -0700

Hi,

I have implemented the JSON-00 draft for YANG-API, and have
some issues that should be addressed.

3.1, para 4:

  The namespace identifier MUST be used for local names that are
   ambiguous, i.e., whenever the data model permits a sibling node with
   the same local name.  Otherwise, the namespace identifier is
   OPTIONAL.

The term "the data model" is unclear.
The module name needs to be used within a protocol
only if the server advertises YANG modules that have
duplicate sibling local-names.

3.1, para 5:

  When mapping namespaces from JSON text to XML, the resulting XML
   document may use default namespace declarations (via the "xmlns"
   attribute), prefix-based namespace declarations (via attributes
   beginning with "xmlns:"), or any combination thereof, following the
   rules stated in [XMLNS].  If prefixed names are used, their prefix
   SHOULD be the one defined by the "prefix" statement in the YANG
   module where each data node is defined.

This draft has no business re-defining XML 1.0,
which says how to encode extended-names.
This paragraph should be removed.

3.2 Mapping XML Elements to JSON Objects

para 1, bullets 1 - 4:

   o  An XML element that is modeled as YANG leaf is translated to a
      name/value pair and the JSON datatype of the value is derived from
      the YANG datatype of the leaf (see Section 3.3 for the datatype
      mapping rules).

   o  An XML element that is modeled as YANG container is translated to
      a JSON object.

   o  A sequence of one or more sibling XML elements with the same
      qualified name that is modeled as YANG leaf-list is translated to
      a name/array pair, and the array elements are primitive values
      whose type depends on the datatype of the leaf-list (see
      Section 3.3).

   o  A sequence of one or more sibling XML elements with the same
      qualified name that is modeled as YANG list is translated to a
      name/array pair, and the array elements are JSON objects.

All 4 bullets mix XML encoding with the YANG construct they
represent.  This does not really work if a node-set is returned
by the protocol:

GET http://localhost/yang-api/datastore/interfaces/interface?select=name

returns:

{
 "data": {
  "name": [
   "eth0",
   "lo"
  ]
 }
}

Even though 'name' is a leaf, since the client requested all
the interface names (eth0 and lo), they are returned as a
leaf-list.  if "select=counters" were used then an array
of JSON objects would be returned (1 per 'counters' container).

Bullet 3 & 4:
I do not agree that arrays MUST be used if only one
leaf-list or list entry is present.  The translator MAY
use an array but not MUST or SHOULD.
The main reason for using JSON
is to get more efficient encoding than XML.

Sec. 3.5 IANA Considerations

It would really help out my code if this encoding was
registered with IANA (MIME type? application/json-yang?)

That way a protocol could identify the encoding types it
supports somehow, and auto-negotiate the one(s) to use.

These are the only implementation issues I had (so far).

Andy

On Tue, Apr 2, 2013 at 10:21 AM, Ladislav Lhotka <[email-masked]> wrote:
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 05 Apr 2013 19:09:45 +0200

> I have implemented the JSON-00 draft for YANG-API, and have

Thanks, I also have a few comments regarding YANG-API, I just have to write them down and send.

> duplicate sibling local-names.

Yes, that's essentially what "data model" means, but I didn't want to limit the mapping to the specific NETCONF context. What one has to know is the complete set of modules, features that are considered active etc. On the other hand, it is true that the hello message provides a very compact way for encoding all this info, so that's why pyangs accepts a hello message as the data model specification.  

So maybe a definition of "data model" along these lines would be appropriate in the terminology section.

> This paragraph should be removed.

This was supposed to mean: An implementation is free to use the default namespace (i.e., no prefix), but *if* it uses a non-default namespace for any part of the data tree, the prefix SHOULD be the one from the YANG module.

> All 4 bullets mix XML encoding with the YANG construct they

I don't think so. We are mapping instance documents, but the way how individidual elements are mapped depends on the data model. 

> of JSON objects would be returned (1 per 'counters' container).

But this cherry-picking breaks the data model schema, so it is outside of this document's scope.

> is to get more efficient encoding than XML.

I disagree, I think it is actually quite useful to be able to encode even single-entry lists as arrays. Consider JSON Pointer. If we have

  container foo {
    leaf-list bar {
      type uint8;
    }
  }

then it is IMO easier to handle if "/foo/bar" always returns an array, and "/foo/bar/0" never results in an error, which would be the case if it was used on

  { "foo": { "bar": 1 }}

And you can predict the behaviour just from the data model, without peeking at the instance document.

> registered with IANA (MIME type? application/json-yang?)

I know there have been flamewars about "media type explosion", but I see no dilemma here because in fact the contents can be *any* JSON text. I don't think the mere fact that there is some data model behind it warrants a new media type.
 
> supports somehow, and auto-negotiate the one(s) to use.

But there is no special encoding, is it?

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 5 Apr 2013 10:26:29 -0700

> This was supposed to mean: An implementation is free to use the default namespace (i.e., no prefix), but *if* it uses a non-default namespace for any part of the data tree, the prefix SHOULD be the…

What difference does it make what prefix is used?
It makes none at all to XML, since an 'xmlns' attribute
MUST be present for whatever prefix is used.
This is a CLR that has no purpose in XML.

> But this cherry-picking breaks the data model schema, so it is outside of this document's scope.

The 'data' node is 'anyxml'.
The bullets above should account for XML elements
which represent YANG anyxml.

> then it is IMO easier to handle if "/foo/bar" always returns an array, and "/foo/bar/0" never results in an error, which would be the case if it was used on

I don't see any added value -- the parser has to accept any
of the valid formats so the code will not prefer 1 form over another.
It will just parse what is there.

> I know there have been flamewars about "media type explosion", but I see no dilemma here because in fact the contents can be *any* JSON text. I don't think the mere fact that there is some data mod…

Obviously I can make up my own URIs, but doesn't
this count as a media type?  It could be application/json
I suppose.

> But there is no special encoding, is it?

The module names and identityrefs, etc. have extra semantics
encoded into the value part.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 5 Apr 2013 20:21:58 +0200

On Apr 5, 2013, at 7:26 PM, Andy Bierman <[email-masked]> wrote:

...

> This is a CLR that has no purpose in XML.

In fact, I agree with you, but the same rule is in RFC 6020, sec. 7.1.4, second paragraph. I just followed the suit.

> The 'data' node is 'anyxml'.

Right, but this anyxml has nothing to do with the ietf-interfaces module.

> It will just parse what is there.

For example, if I want append a new entry (say 3) at the end, I can always use this JSON Patch operation:

  { "op": "add", "path": "/foo/bar/-", "value": 3 }

It works fine for both {"foo": {"bar": [1, 2]}} and {"foo": {"bar": [1]}}, but fails for
{"foo": {"bar": 1}}.

> this count as a media type?  It could be application/json

Or perhaps something specific for the concrete data model. Maybe we can ask application experts.

Lada

---

From: Phil Shafer <[email-masked]>
Date: Sat, 6 Apr 2013 13:56:21 -0400

Ladislav Lhotka writes:
>1. I don't know how to encode the binding between such an attribute and the data node (e
>specially leaf) to which it belongs, because in JSON they would have to be siblings, and
> there might be other sibling leaves.

XML data has three dimensions, values, attributes, and namespaces.  If
you have to encode using JSON, you'll need something like:

    {
        "top": {
            "_attributes": {
                "elt1": {
                    "attr1": "av1",
                }
                "elt2": {
                    "attr2": "av2",
                     "more2": "mv2",
                }
            },
            "elt1": "25",
            "elt2": "popcorn",
        }
    }

for the XML:

    <top>
        <elt1 attr1="av1">25</elt1>
        <elt2 attr2="av2" more2="mv2">popcorn</elt2>
    </top>

Noisy as all get out, but....

><module name>:<local name>

Ah, I missed that you are talking about an actual module name here.
It wasn't clear that you are talking about the name of the YANG module.
Once I know what you mean, it's clear, but adding something more explicit
might help the next reader.

I'm also worried about this part:

This is both confusing (since you're talking about "namespace identifier"s
instead of module names, but it's also painful for both parties to
not know if the other side is sending the module name or not.  My
code can't refer to 'top.elt1' or 'top["elt2"]' since they could be
top["phil:elt1"] or top["phil:elt2"].

Ambiguity is also an interesting point, since deciding what's ambiguous
means knowing the full set of modules running on the device.  If you
add the "lada" module, then top.elt1 has to be top["phil:elt1"] if
"lada" augments with elt1.

>may lead to local name collisions where explicit namespace ids are required. But if the 
>local name is unambiguous, the <module>: prefix is not required, even for augmented node
>s.

I still suggest an example since this is not an easy topic and is
only covered lightly in the text.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Sat, 6 Apr 2013 11:28:42 -0700

> you have to encode using JSON, you'll need something like:

Qualified attributes could be encoded like node names.
I prefer to keep the attributes with the parent nodes to make
parsing, streaming, and access control easier.

There could be a convention that "@attributes" and "@value" must
be used together:

   { "config" : {
      "newleaf": {
         "@attributes": {
            "ietf-netconf:operation" : "create"
         },
         "@value" : 42
      }
  }

This works for any YANG data node (replace 42 with
appropriate conversion according to draft).
I think @attributes is a JSON convention already,
and it is an illegal YANG identifier, so no name
collisions are possible.

> might help the next reader.

I agree -- it is up to the protocol that uses this encoding to
provide a mapping from module-name to a more specific
description of the relevant YANG module file(s).

>>local name is unambiguous, the <module>: prefix is not required, even for augmented node

This draft needs to assume a "static" YANG module, meaning
all the augment-stmts are already applied.  If the set of modules
on the server has duplicate local-names due to augment or
just 2 top-level nodes from different modules, then the "long-form"
of the name has to be used.

> only covered lightly in the text.

agree -- encoding examples for all YANG node types and data types
are needed.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 07 Apr 2013 12:51:27 +0200

> you have to encode using JSON, you'll need something like:

Hmm, this is really ugly and I think it sort of defeats the purpose of using JSON in the first place, i.e. simple encoding. IMO, it would be better to keep the attributes completely separate, for example

    [
      {
        "path": "/top/elt1",
        "attributes": { "attr1": "av1" }
      },
      {
        "path": "/top/elt2",
        "attributes": { "attr2": "av2", "more2": "mv2" }
      }
    ]

Another approach could be to turn things around:

  "@disabled": {
    "elt1": 25,
    "elt2": "popcorn"
  }

The advantage of this approach is that it works for leaf-list entries as well:

  "foo": [1, 2, {"@disabled": 3}, 4]

But all that being said, the draft is about mapping documents that follow the schema of a YANG data model, and YANG cannot model XML attributes, so they are out of scope.

> might help the next reader.

I agree that the text can be improved and one or two examples would be useful.

> instead of module names, but it's also painful for both parties to

Module name is the namespace identifier.

> top["phil:elt1"] or top["phil:elt2"].

We have gone through several iterations here, and I think the present solution is best (it was Andy's suggestion). The motivation was to get rid of namespaces where possible. It's true that this simple access to object members is not possible but a simple function would do. The alternative would be to have module names everywhere.

> "lada" augments with elt1.

Yes, it is a basic assumption that all modules and features etc are known beforehand.

> only covered lightly in the text.

OK, agreed.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 07 Apr 2013 13:14:25 +0200

> and it is an illegal YANG identifier, so no name

Perhaps the problem is that YANG-API is just a front end to NETCONF API. NETCONF operations were designed for XML and don't work that well for JSON.

IMO, using JSON Patch (it is now a fresh RFC 6902) would be simpler and better:

  [
    { "op": "add", "path": "/config/newleaf", "value": 42 }
  ]

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sun, 7 Apr 2013 07:11:34 -0700

> Perhaps the problem is that YANG-API is just a front end to NETCONF API. NETCONF operations were designed for XML and don't work that well for JSON.

YANG-API does not use attributes.
If I want to use an alternate encoding for NETCONF, then
attributes need to be supported.  I've seen several examples
of how this is being done and it doesn't look too complicated.

>     { "op": "add", "path": "/config/newleaf", "value": 42 }

This actually has a different edit model than <edit-config>.
The PATCH operation is atomic, and each individual patch
object within the request is sequential.  They expect the
input for step 3 to be the output of step 2.

Rather than validating N separate edits against candidate or running.
JSON Patch validates edit N+1 against result N.
Of course if any operation in the list fails (including 'test'),
then everything has to get backed out, even on the running
config (rollback-on-error is the only error-option).

I don't know if this edit model is compatible with the NETCONF
datastores.  If the only thing going on were simple patches
to a document, then it would be easy.  But a NETCONF
datastore is the tip of the iceberg -- the underlying system
takes those edits and changes network behavior.
It could be impossible to to know if result N will be valid
until an attempt to commit that change is made.
This has always been the flaw with pretending the datastore
is just an XML instance document.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 7 Apr 2013 17:19:31 +0200

> is just an XML instance document.

Validation always has some scope and limits. A configuration that is perfectly OK from the point of view of an individual router can still break the network. 

But I think that RFC 5789 (HTTP PATCH) addresses all kinds of errors, even run-time. 

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sun, 7 Apr 2013 09:15:21 -0700

> Validation always has some scope and limits. A configuration that is perfectly OK from the point of view of an individual router can still break the network.

This has nothing to do with the problem.

> But I think that RFC 5789 (HTTP PATCH) addresses all kinds of errors, even run-time.

I think the NETCONF model works better for configuration management.
There is a huge difference between JSON Patch and NETCONF:
  * NETCONF applies N edits within 1 request in no specified
     order to the target datastore, and all validation is against the target
  * JSON Patch applies N edits within 1 request in sequential order.
     The first edit is validated against the target which produces a temporary
     result datastore R(n).  Each edit(n+1) is validated against datastore R(n).

I suppose only server developers understand the impact of this change,
but IMO the NETCONF edit model is much better for CM use-cases
and much easier to implement.

I am interested in maintaining compatibility with NETCONF servers
with any new protocol or message encoding.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 7 Apr 2013 19:19:17 +0200

>      result datastore R(n).  Each edit(n+1) is validated against datastore R(n).

Andy, can you point us to where this is specified for JSON patch? Do
they really have the same notion of 'validation' that NETCONF has?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Sun, 7 Apr 2013 11:54:41 -0700

> they really have the same notion of 'validation' that NETCONF has?

They have a notion that the operations are performed sequentially
and value tests can be done against intermediate steps (RFC 6902, sec. 3).
Even if just the final result is validated wrt/ datastore checks, the field
validation and the individual "test" operations within the patch may
need to be done on intermediate results.

The server should be the one that figures out the order and
normalization of edits.  I prefer the current NETCONF edit
model and see no reason to change it.  I'm just looking for more
efficient message encoding formats than XML, not a server re-design.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 7 Apr 2013 22:04:14 +0200

> efficient message encoding formats than XML, not a server re-design.

Do we actually have evidence that applications out there make use of
the NETCONF edit model? There is complexity associated with the
ability to send an arbitrarily complex edit-config to a device. If
this is heavily used, fine. If not (e.g., since application writers
are too lazy to generate complex edit-configs or they simply only
operate on a small subset of the data model anyway), then one can ask
whether maintaining this complexity for a REST API is a good match.

Note, this is a question. I do not know the answer.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 07 Apr 2013 23:18:53 +0200 (CEST)

Juergen Schoenwaelder <[email-masked]> wrote:
> ability to send an arbitrarily complex edit-config to a device.

We make heavy use of it!  The complexity shows up when we have to deal
with devices that do *not* support this model.  Given an arbitrary
changeset, figuring out which CLI commands / SNMP SETs / REST
PUT,POST,PATCH you have to send, and most importantly, in which
order to send them is terribly tricky in the general case.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 07 Apr 2013 23:20:29 +0200 (CEST)

> need to be done on intermediate results.

I do not understand this.  Why can't we say that the final result is
validated and applied to the datastore?  What is the problem with
field validation and "test"?

> efficient message encoding formats than XML, not a server re-design.

+1

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 8 Apr 2013 00:17:10 +0200

> order to send them is terribly tricky in the general case.

But this is likely not what this discussion is about. I want to
understand why a NETCONF edit-config with embedded "add this", "merge
this", "delete this" and then you validate the result is fundamentally
better than a JSON patch that you send and then validate the result.

I heard Andy saying that an edit-config can be processed in any order
the server likes and that is a big win. However, applying a JSON patch
yields another JSON document that can be validated and then the delta
applied to the system in any order as well, no? We may have to
distinguish between (a) the transformation of the serialized
representation and (b) the application of the change to the system.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 08 Apr 2013 00:23:50 +0200 (CEST)

Juergen Schoenwaelder <[email-masked]> wrote:
> But this is likely not what this discussion is about.

Ok, then I got confused by your question.  I guess I don't understand
what your question refers to then; specifically, what does this mean:

  There is complexity associated with the ability to send an
  arbitrarily complex edit-config to a device.

> I want to
> applied to the system in any order as well, no?

Yes, that is also my understanding.  I think Andy needs to clarify his
concerns.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sun, 7 Apr 2013 15:24:28 -0700

> field validation and "test"?

The "test" operation is an equals-test on the previous result in the sequence.
It is not a test on the end-result.

The JSON Patch edit list can be a random sequence of edits,
add then delete, then add the same entry, set the value
to 10, test for 10, set the value to 20, test for 20, etc.

I guess if the server looked at the edit as being only
the final result of the atomic PATCH operation and
recorded edits according to the diffs it finds and applies the edits
to the system any way it wants, then this would be OK.

The server does not have to preserve intermediate edits
in the sys-config-change notifications.  E.g., a 2 edit patch like
"add /top/a; remove /top/a" can result in a NO-OP on the
server.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 8 Apr 2013 10:14:46 +0200

> to the system any way it wants, then this would be OK.

Yes, this is how I tried to look at things.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 08 Apr 2013 11:23:24 +0200

> This has nothing to do with the problem.

I think it does have, in the sense that it is up to the server to accept or reject a request, be it for a stupid typo or very tricky run-time error.

>      result datastore R(n).  Each edit(n+1) is validated against datastore R(n).

This is not necessarily true. HTTP PATCH requires the patch request to be atomic, so that it is either applied or rejected in its entirety, and a partially patched resource must not be served to anybody else. I don't see anything that prevents the server to perform just a single validation on the final patched contents. Of course, other errors may occur in the middle of JSON Patch processing (including failed "test" operation).

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 08 Apr 2013 11:25:46 +0200

> order to send them is terribly tricky in the general case.

But isn't it because there is no standardized API?

Lada

---

From: Phil Shafer <[email-masked]>
Date: Wed, 10 Apr 2013 19:09:35 -0400

>parsing, streaming, and access control easier.

It's a tradeoff.  Having attributes with the parent node makes it
easier to get to the attributes at the expense of making it more
painful to get to the real values.

>            "ietf-netconf:operation" : "create"

This turns the value from config.newleaf to config.newlead["@value"]
(in javascript) which is decidedly less appealing.

>of the name has to be used.

But the client won't know if the server needs to use the
"long form" unless it completely groks all the modules that
server implements, which isn't realistic.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Wed, 10 Apr 2013 19:13:56 -0400

Ladislav Lhotka writes:
>But all that being said, the draft is about mapping documents that follow the schema of 
>a YANG data model, and YANG cannot model XML attributes, so they are out of scope.

I don't see how this can work.  YANG defines means of manipulating
the data defined in its data models.  If you can't express that,
you've either dramtically reduced your usefulness or are needing
to redefine all the tools for manipulation.  Both are non-starters
imho.

>Yes, it is a basic assumption that all modules and features etc are known beforehand.

How can this assumption be valid in the "wild", where precious
little is known beforehand?

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Wed, 10 Apr 2013 19:16:38 -0400

>order to send them is terribly tricky in the general case.

Ditto and amen.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Apr 2013 08:31:38 +0200

> the data defined in its data models.  If you can't express that,

The purpose of this document is to enable encoding of datastore contents (or other YANG-modelled data trees) in JSON and their validation against a YANG data model. As such, it works just fine.

> to redefine all the tools for manipulation.  Both are non-starters

Not necessarily, there are other means for specifying edits, such as JSON Patch.
 
> little is known beforehand?

Why? Servers know their data model and send all this info in hello, and REST approaches will presumably make it available through a well-known URI.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Apr 2013 06:40:53 -0700

> Not necessarily, there are other means for specifying edits, such as JSON Patch.

There are also ways to encode attributes, even if we don't
agree on the details yet.

I agree that JSON Patch is better than using attributes
like operation=create or insert=after.

> Why? Servers know their data model and send all this info in hello, and REST approaches will presumably make it available through a well-known URI.

I agree.  There are several companies that have tools which
utilize the hello, get-schema, and /netconf-state info and
build the exact schema tree that the server advertises, starting
from scratch.

That doesn't mean the tool knows the semantics of the knobs,
but 100% of the YANG and CRUD mechanics can be automated.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Thu, 11 Apr 2013 10:30:34 -0400

Ladislav Lhotka writes:
>The purpose of this document is to enable encoding of datastore contents (or other YANG-
>modelled data trees) in JSON and their validation against a YANG data model. As such, it
> works just fine.

Skimming the JSON patch draft; looks like it's missing some vital
functionality.

But in truth, this JSON/REST work is falling into the "netconf 2.0"
trap, where every is being redesigned.  If so, perhaps these issues
don't concern you.

For me, the keys are having a stable usable technology that folks
can implement and deploy.  Anything that implies a redesign is a
step in the wrong direction.

>Why? Servers know their data model and send all this info in hello, and REST approaches 
>will presumably make it available through a well-known URI.

One of the ideas in NETCONF/YANG is that the client doesn't need
to completely understand everything the server implements.  If you
support the FOO model, then my client needs only to grok that model
in order to manipulate your box.  Forcing the client to have complete
understanding would be a step backwards.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Thu, 11 Apr 2013 10:42:07 -0400

>like operation=create or insert=after.

I must be missing something.  How does one get insert=after
functionality from JSON patch?  "add" takes only a simple path,
as does "move".

Maybe this is a problem solved by json schema?  Or json pointer?
Or by any of the "redo stuff we had in xml for json" drafts?

I'm still trying to understand what technological improvement is
made when we say '"foo": "none",' instead of '<foo>none</foo>'.
I'm just not seeing it.....

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Apr 2013 17:09:45 +0200

> functionality from JSON patch?  "add" takes only a simple path,

   o  If the target location specifies an array index, a new value is
      inserted into the array at the specified index.

> I'm just not seeing it…..

Of course, the problem with XML is not in a different serialisation but rather in its complexity - mixed content, namespaces etc. Generic tools have to support all this.

But nobody says XML is evil, JSON is just an alternative.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Apr 2013 17:28:38 +0200

> Skimming the JSON patch draft; looks like it's missing some vital

It's simpler but I think everything what edit-config does should be doable, too, perhaps with a few more round-trips. On the other hand, JSON Patch clearly separates the locator ("path") from the new contents ("value"), which is a good thing.

> step in the wrong direction.

NETCONF notwithstanding, folks keep designing REST APIs for configuration purposes, even in the IETF. I think it would be great if YANG could work for them, too.

> understanding would be a step backwards.

Such a client with an incomplete model can always use the module: prefix with all node names, this is allowed. Our goal was to get rid of explicit namespace identifiers where possible.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Apr 2013 11:53:18 -0700

>> step in the wrong direction.

I am still trying to understand the JSON Pointer RFC,
but I think using array position for edit operations
may be incompatible with system-ordered lists and leaf-lists.

RFC 6020 is not clear whether system ordered means
the server picks some ordering and uses the same ordering
every time, or whether it means there is no document
ordering that the client can depend on.

I think the recent interpretation of JSON Patch implies that
the document ordering has to be stable for the entire PATCH
operation.  That isn't good enough, which is why the If-Match
header is needed.  The entity tag must be changed if document
order of the conceptual datastore or subtree changes.

> NETCONF notwithstanding, folks keep designing REST APIs for configuration purposes, even in the IETF. I think it would be great if YANG could work for them, too.

I am only interested in using JSON Patch in YANG-API, not NETCONF.

> Such a client with an incomplete model can always use the module: prefix with all node names, this is allowed. Our goal was to get rid of explicit namespace identifiers where possible.

I don't see how JSON or REST changes anything related to this problem.

Andy

---

From: t.petch <[email-masked]>
Date: Fri, 12 Apr 2013 10:16:30 +0100

----- Original Message -----
From: "Andy Bierman" <[email-masked]>
To: "Ladislav Lhotka" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, April 11, 2013 7:53 PM
> > It's simpler but I think everything what edit-config does should be
doable, too, perhaps with a few more round-trips. On the other hand,
JSON Patch clearly separates the locator ("path") from the new contents
("value"), which is a good thing.

Andy

I don't know if you ever look at apps-discuss but JSON in its various
aspects has figured substantially there in the past year or two, and
every time it does, it seems to me like a 'topsy' design, lacking a
coherent underpinning, with weak typing and much coercion. Whatever you
do will produce something (as opposed to an error) but you may be
surprised at what that is.

 Recent examples of posts about JSON Pointer  include

'If we were starting from scratch and defining JSON Pointer again I
would argue for distinguishing array indices and object names in the
syntax. For instance, prefix an object name with "/" and an array index
with ":". '

'JSON Pointer does not distinguish between objects and arrays. That is
not
determined until the pointer is applied to an actual object instance...
the
pointer "/1" is valid against {"1":"a"} or ["a","b"] '

'If I understand the issue
correctly, you're basically talking about a client making a blind
partial
update to an arbitrarily structured JSON document whose structure may
have
changed since the last time the client edited it...  In such cases, I
would
argue that you get whatever you end up with.'

This last says to me that you must know precisely what you have to start
with or else you will have no idea what you will end up with, ie lock
the document, read it in its entirety, apply an update, release the
lock - anything else is unpredictable.

Overall I have the impression that there are a lot of flaws yet to be
discovered in the JSON specifications and they are something I hope that
I never have to be involved with.

If you find it hard to understand, perhaps it is because you are asking
questions of it that do not yet have answers:-(

Tom Petch

> > NETCONF notwithstanding, folks keep designing REST APIs for
configuration purposes, even in the IETF. I think it would be great if
YANG could work for them, too.
> > Such a client with an incomplete model can always use the module:
prefix with all node names, this is allowed. Our goal was to get rid of
explicit namespace identifiers where possible.

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Apr 2013 11:44:43 +0200

> ordering that the client can depend on.

Well, if it is system-ordered, then why bother? New entries can be always appended at the end, right?

> I am only interested in using JSON Patch in YANG-API, not NETCONF.

Sure.

> I don't see how JSON or REST changes anything related to this problem.

In XML, the namespace is always expressed, via NS declarations. In JSON, it needn't be the case. If the server advertises modules A and B, and the client only cares about A, then there may be a local name clash that the client is not aware about, caused by an augment in module B.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Apr 2013 13:05:24 +0200

> syntax. For instance, prefix an object name with "/" and an array index

This is not a big issue for us because, luckily, we have a data model, so we are able to discriminate between containers/objects and lists/arrays.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 04:54:00 -0700

> Well, if it is system-ordered, then why bother? New entries can be always appended at the end, right?

No - YANG insert on a system-ordered list is an error.
Strictly speaking, an array insertion on any system-ordered
object is always an error.

Either the server honors the user request and treats the
data as user-ordered, or the server lies (returns OK)
and inserts the data wherever it wants, or the server
returns an error.

> In XML, the namespace is always expressed, via NS declarations. In JSON, it needn't be the case. If the server advertises modules A and B, and the client only cares about A, then there may be a loc…

The client will get back a qualified name whether it knows
about the other module or not. If it chooses to ignore all
qualified names that don't match A, then so what?
Garbage in = garbage out.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 05:02:43 -0700

> surprised at what that is.

Perhaps they should be using data models instead.
One key feature of Lada's draft is that the underlying
schema drives the CRUD operations, not the
instance document.  A container does not magically
become a list as soon as there are 2 of them.

I think YANG insert + point approach is much better
than JSON Patch because it does not rely
on document position.

> lock - anything else is unpredictable.

The server needs to implement entity tags correctly
and the client needs to use IfMatch <etag> correctly.
The server will send a 412 PreCondition Failed error
if the client is not editing the version they think they are editing.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Apr 2013 14:54:24 +0200

> object is always an error.

I cannot find any text in 6020 saying this is an error.

> and inserts the data wherever it wants, or the server

We can imagine the server reshuffles the entries of an system-ordered list after an insertion, so the actual array index (including '-') is irrelevant and the result will be always the same.

…

> Garbage in = garbage out.

Yes, but in the opposite direction, the client can send a name with no namespace, and it will be ambiguous for the server. 

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Apr 2013 15:10:29 +0200

> become a list as soon as there are 2 of them.

Exactly, and that's why I also insist on single-entry lists being still represented as arrays.
 
> than JSON Patch because it does not rely

Can you give an example?

On the other hand, JSON Patch can easily do things that are forbidden in edit-config, such as change a list key. This is because the locator ("path") is detached from "value".

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 06:29:18 -0700

> I cannot find any text in 6020 saying this is an error.

I cannot find any text that says the insert attribute
refers to something other than a user-ordered list.
All the text related to the insert operation specifically
mentions "user-ordered lists".

IMO this was the intent of the WG.

7.7.1, para 4:

   YANG provides a rich set of facilities within NETCONF's <edit-config>
   operation that allows the order of list entries in user-ordered lists
   to be controlled.  List entries may be inserted or rearranged,
   positioned as the first or last entry in the list, or positioned
   before or after another specific entry.

7.7.7, para 2

   In an "ordered-by user" leaf-list, the attributes "insert" and
   "value" in the YANG XML namespace (Section 5.3.1) can be used to
   control where in the leaf-list the entry is inserted.  These can be
   used during "create" operations to insert a new leaf-list entry, or
   during "merge" or "replace" operations to insert a new leaf-list
   entry or move an existing one.

7.8.6, para 2

   In an "ordered-by user" list, the attributes "insert" and "key" in
   the YANG XML namespace (Section 5.3.1) can be used to control where
   in the list the entry is inserted.  These can be used during "create"
   operations to insert a new list entry, or during "merge" or "replace"
   operations to insert a new list entry or move an existing one.

> We can imagine the server reshuffles the entries of an system-ordered list after an insertion, so the actual array index (including '-') is irrelevant and the result will be always the same.

I thinkt it as an error.
Otherwise the server is forced to treat an unordered list
as ordered for the duration of the PATCH operation.
Each edit in the patch list is applying or testing
the document based on the expected document order.

The YANG schema needs to drive the validation.
These rules are specified by the YANG designer,
not the client sending a JSON Patch.

It is usually extremely expensive for the server to treat
all its managed data as a static instance document.
JSON Patch is designed to work on documents not datastores.
Although it could be forced to work on a datastore,
the implementation and runtime resource cost would be significant.

> Yes, but in the opposite direction, the client can send a name with no namespace, and it will be ambiguous for the server.

So the server will send an error and the client
will be forced to retry using namespaces.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 12 Apr 2013 15:35:12 +0200 (CEST)

> IMO this was the intent of the WG.

Yes, and I think the text is clear.

[...]

>    control where in the leaf-list the entry is inserted.

Here is it clear that the attributes can be used for ordered-by user
leaf-list.  We do not everything that it _cannot_ be used for,
e.g. container, notification, rpc, input, ...

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 06:37:47 -0700

...
>> than JSON Patch because it does not rely

NETCONF (or YANG-API) does not allow
the client to make a new Nth list entry.
You have to identify the insertion point by name.
No 2 servers are expected to use the same
system ordering.  It is not even clear if a server
must maintain the same system order from
1 message to the next, let alone across a reboot.

Insertion by document order is full of problems
and writing reusable client code based on it is a fool's errand.

> On the other hand, JSON Patch can easily do things that are forbidden in edit-config, such as change a list key. This is because the locator ("path") is detached from "value".

This is a conceptual "remove" followed immediately
by an "add" operation.  I think YANG insert is
overloaded and a separate "move" attribute would be better.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Apr 2013 16:11:11 +0200

On Apr 12, 2013, at 3:35 PM, Martin Bjorklund <[email-masked]> wrote:

> e.g. container, notification, rpc, input, …

Actually, sec. 8.3.1 does say something:

   o  If the attributes "before" and "after" appears in any element that
      is not a list whose "ordered-by" property is "user", the server
      MUST reply with an "unknown-attribute" error-tag in the rpc-error.

This seems incorrect, it should be 'If the attribute "yang:insert" whose value is either "before" or "after" appears …'. But even then, it says nothing about "first" and "last".

So maybe what was really intended was 'If the attributes "yang:insert" or "yang:value" appear …'?

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 07:31:17 -0700

> So maybe what was really intended was 'If the attributes "yang:insert" or "yang:value" appear …'?

No, Martin is correct. Only "ordered-by user" nodes can use
the "insert" attribute.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Apr 2013 16:33:08 +0200

> 1 message to the next, let alone across a reboot.

Mikrotik/RouterOS has an interesting solution to this: The output of "print" command presents list entries in a certain order (and assigns numbers to them), and this numbering is guaranteed for that CLI session till the next "print". I guess something like this would be necessary here, too.

Another option is to use more powerful query expressions instead of JSON Pointer, e.g. like those in MongoDB.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 12 Apr 2013 16:34:58 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> This seems incorrect, it should be 'If the attribute "yang:insert"
> whose value is either "before" or "after" appears …'.

You are right, this text is not correct.  There is no "before"
attribute.  I will check the text.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 12 Apr 2013 08:03:29 -0700

>>>> than JSON Patch because it does not rely

Here is a better example why using document order is
a really bad idea when managing networks.

What if the JSON Patch edit on an array is a route injection
into config or operational state on a large router?
The "data nodes" are not represented in YANG trees
within the router.  They are probably not even all in
1 internal data structure, but spread out in various ways.

Asking the router to freeze document order and process
edit or retrieval requests for the Nth route entry is a non-starter.
(e.g, N=42003).  The router needs data such as prefix/length
to know where in the system to find the internal table.
There is no document order maintained in the router for this list.
This is an alien and irrelevant concept within the router.

> Another option is to use more powerful query expressions instead of JSON Pointer, e.g. like those in MongoDB.

I want to leverage existing NETCONF code, so I am
not inclined to reinvent NETCONF datastore editing
just to use JSON message encoding. IMO the message
encoding needs to be completely detached from the
model (classic MVC stuff).  In fact, there is no reason
the server could not support extensible message formats
and let vendors or operators install plugins that support
whatever latest-and-greatest encoding they want to use.

Andy