Subject: [netmod] augment by revision

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 16 Nov 2011 10:18:24 +0800

Hi,

before adding more revision complexity, how can we resolve the following situation?

Module A imports B in revision X and augments some of its nodes. Then, a server advertises in hello module A and B in revision Y.

Does it mean that the augments won't be applied?

Lada

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Nov 2011 09:37:56 +0100 (CET)

Hi,

Ladislav Lhotka <[email-masked]> wrote:
> Module A imports B in revision X and augments some of its nodes. Then,
> a server advertises in hello module A and B in revision Y.

I think this is illegal.  The server is supposed to advertise all
modules it implements.  If it faithfully implements A, it follows that
it also implements B in revision X.  Thus it must also advertise B in
revision X.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 16 Nov 2011 16:59:08 +0800

> it also implements B in revision X.  Thus it must also advertise B in

Well, but it can happen quite easily: Module A fixes a bug, so a server developer upgrades to the new version without realizing that module A also "upgraded" the import of B. If you have several dozen interconnected modules, this would be pretty difficult to spot.

How about the same situation where module A doesn't augment anything, just uses B's groupings and typedefs. 
So potentially different revisions of groupings/typedefs are used in A and in B. Would this be legal?

If hello advertised submodules with revisions, we'd have a similar situation: what if hello advertises  a submodule revision that is different from the revision included by the main module? 

In all cases, the problem is the same: certain parts of the data model are specified in two places, so inconsistencies may occur.

Lada

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Nov 2011 10:09:44 +0100 (CET)

Ladislav Lhotka <[email-masked]> wrote:
> Well, but it can happen quite easily: Module A fixes a bug, so a
> server developer upgrades to the new version without realizing that
> module A also "upgraded" the import of B. If you have several dozen
> interconnected modules, this would be pretty difficult to spot.

If it is illegal it is illegal.  If you're using a tool like yuma or
confd, this can't happen.  If you roll your own <hello> message you
have to make sure you follow the spec.

> and in B. Would this be legal?

Sure.

> If hello advertised submodules with revisions, we'd have a similar
> situation: what if hello advertises a submodule revision that is
> different from the revision included by the main module?

I think this should be illegal.  In fact, I don't think the server
should have to advertise a submodule that is included by revision.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 16 Nov 2011 17:43:16 +0800

> have to make sure you follow the spec.

OK. In fact, my recent extension to pyang allows to submit a hello message, so it could be checked there, too. But what would you actually check? It cannot simply be that module A imports a different revision of B than is the one advertised in hello.

>> and in B. Would this be legal?

Hmm, this could lead to really nasty surprises. Imagine that B=ietf-interfaces and its data tree is changed in revision X so that the path in the definition of "interface-ref" changes. Then this path must be broken in either A or B/Y, which together form the data model advertised in hello.

> should have to advertise a submodule that is included by revision.

Isn't it then simpler to always include by revision?

Lada

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 16 Nov 2011 10:56:21 +0100 (CET)

> Isn't it then simpler to always include by revision?

This goes back to my question if advertising submodule revisions is
the correct solution.

So, the problem we are trying to solve is that you want to have
separate pieces of a data model for one namespace, each piece
individually versioned.

Today, using submodules to solve this won't work, b/c if you use
include by revision, you can't mix these pieces in arbitrary ways, and
if you don't use include by revision, you don't know which versions
the server implements.

Also, using modules won't work, since each module is tied to one
unique namespace.

One solution to this problem is to advertise submodule revisions.

Another solution to this problem is what Kent suggested; use modules,
which already are versioned and advertised individually, but allow
multiple modules to share the same namespace.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Nov 2011 02:30:01 -0800

On 11/16/2011 01:56 AM, Martin Bjorklund wrote:
> Ladislav Lhotka<[email-masked]>  wrote:

We don't have any control over what vendors use -- just the IETF.
IMO, submodules are not very usable because there is only 1 revision for the
entire module.

I do not agree that we have a problem with too many XML namespaces.
I do not understand why this would make the YANG documents more
or less readable.  I do not care about people reading raw XML off the wire.
That is not an important use-case.  It is not difficult to read XML that uses
namespaces.

I find the current NETMOD ip and snmp-cfg drafts hard to read because
there is 5X the boilerplate overhead and I have to jump back and
forth between submodules constantly to read them.

> /martin

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 16 Nov 2011 19:01:25 +0800

> Also, using modules won't work, since each module is tied to one

In accord with Andy, I don't see the number of namespaces as a threat to readability.

> multiple modules to share the same namespace.

Both solutions go in the direction of using module names and revisions as de facto namespace discriminators, which is OK but makes me wonder why we bother with XML namespaces at all.

Lada

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2011 13:54:15 +0100

> In accord with Andy, I don't see the number of namespaces as a threat to readability.

I disagree. Complexity is bad. And if can avoid it, we do make life
simpler. And this is in particular true for organizations like the
IETF were refactoring is hard to impossible and work is sliced into
pieces according to procedural aspects and learning processes. I know,
all this is debatable. But the argument "you just have to have the
right tools" does not convince given the experience in SNMP where the
MIB browser still is a common interface.

That said, let me bring up technical arguments. ;-) Here is a simple
task: Write an <get-config> filter that retrieves all NETCONF
configuration objects from the various modules we have in the
pipeline. Compare that to a filter if everything would be in a shared
namespace.

I believe there is a big value in being able to retrieve everything
beloning to function X with a simple filter without having to know
whether the IETF did split this in 10 or 20 separate modules.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Nov 2011 05:37:35 -0800

On 11/16/2011 04:54 AM, Juergen Schoenwaelder wrote:
> task: Write an<get-config>  filter that retrieves all NETCONF

NETCONF does not have any filtering by a specific XML namespace.
RFC 6241 added a subtree wildcard mechanism that matches any namespace.
Simply requesting the top-level node will return all descendants, regardless
of namespace.  In my experience, browsers like firefox have no problem
at all displaying XML with namespaces.

IMO, submodules increase complexity since they reduce readability and their
revisions are not advertised in the <hello>.  It is also difficult to follow
submodules back to their parent revision, since the belongs-to-stmt
has no revision-date, and include-by-revision is optional-to-use.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 16 Nov 2011 22:21:07 +0800

> MIB browser still is a common interface.

I don't think XML namespaces by themselves increase complexity, they are just a general solution to a real problem. Their main drawback is in the disconnect between the NS URI and the prefix but this doesn't apply to YANG, because we have the URI and prefix neatly defined in one place and both are IANA-registered for IETF modules.

IMO the complexity stems from the fact that we have several overlapping parameters that define the meaning of names:
- module name
- prefix
- namespace URI
- revision date

And also that they may occur both in YANG modules, instance documents and hello messages.

I can imagine a scheme where node names have no XML namespace at all, for example

<if.interfaces yang-module="ietf-interfaces">
  …
</if.interfaces>

And something like this would equally well work e.g. in JSON (hence, JETCONF:-).

But it is too late for this.

> whether the IETF did split this in 10 or 20 separate modules.

This rather suggests that NETCONF subtree filtering is lame and not fit for its purpose.
Such a task is a piece of cake for XPath/XSLT.

Lada

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2011 15:25:57 +0100

On Wed, Nov 16, 2011 at 05:37:35AM -0800, Andy Bierman wrote:
 
> of namespace.

Is this also true for xpath expressions? BTW, what is the common
toplevel in the case of NETCONF data models? ;-)

> In my experience, browsers like firefox have no problem

Nobody said browsers would be a problem. I am talking about simplicity
at the human interface - having to recall or lookup that some feature
that got added to protocol X two years later sits in a different
namespace is something worth avoiding.

> IMO, submodules increase complexity since they reduce readability
> and their revisions are not advertised in the <hello>.  It is also
> difficult to follow submodules back to their parent revision, since
> the belongs-to-stmt has no revision-date, and include-by-revision is
> optional-to-use.

Yes, that is what we are discussing here. I have seen the following
options so far:

a) Declare submodules are broken, do not use them and we assume that
   users have tools that make it easy for them to deal with hundreds
   of namespaces.
b) We declare that submodules should only be used with 
   include-by-revision, although that limits flexibility.
c) We make submodules usable by announcing them including their revisions
   in the <hello> exchange.
d) We declare submoduldes are broken and instead accept that multiple
   modules silently can share the same namespace (and give up the
   ability to check at module compile time that there are no name clashes).

My preference so far is c).

/js

---

From: Phil Shafer <[email-masked]>
Date: Wed, 16 Nov 2011 11:05:15 -0500

>   ability to check at module compile time that there are no name clashes).

e) allow module owners to force submodules to use import-by-revision,
where upon submodule revisions would be announced in the <hello>
message.

I definitely do not want to lose the existing ability to take a
single module and divide it into submodules so developers don't all
work in the same file.  We have >300 submodules with scores of
developers and need submodules for organizational sanity.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Nov 2011 11:00:09 -0800

On 11/16/2011 08:05 AM, Phil Shafer wrote:
>>    in the<hello>  exchange.
> where upon submodule revisions would be announced in the<hello>
> message.
>
> work in the same file.  We have>300 submodules with scores of

You seem to be referring to server developers.
I am also concerned about the usability for client application developers.

At the interim (way back) when these things were designed,  I expressed
concern that we we recreating the mistakes of AGENTX. We were promised
that sub-agents would be completely transparent to the NMS, but that was not true.

In a way, submodules are transparent to the client, because only the parent module
name/version is known.   We could decide that they are behaving as intended.
Submodules are not manageable by the client.  There is only 1 name/version
for the module, whether it contains a few objects or the whole kitchen sink.
Since the main module revision-date is bumped every time any submodule
is added or changed, the client won't be able to use the revision-date
to determine which submodules may potentially be affected.

Then there is the issue of conformance.  You can obsolete a submodule.
Does that make the entire module obsolete?   Can this be easily managed
by the client without proper visibility into sub-module lifecycle details?

Lada brings up some valid concerns about design complexity we created
by the way we use names, prefixes, and namespaces.  That is the origin of the
true complexity, not XML namespaces themselves.

I agree with Phil that our original intent was that submodules be just as usable
as modules, but we never really specified what that meant.
I suggest before we debate solutions, we agree on the problems first.

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 16 Nov 2011 12:31:44 -0800

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Phil Shafer" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, November 16, 2011 11:00 AM
> Subject: Re: [netmod] augment by revision
...
> that sub-agents would be completely transparent to the NMS, but that was not true.

Out of curiosity, just how are AgentX subagents not transparent
to an NMS in any way that matters?

Randy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Nov 2011 14:07:13 -0800

On 11/16/2011 12:31 PM, Randy Presuhn wrote:
> Hi -
>
>> From: "Andy Bierman"<[email-masked]>
>> To: "Phil Shafer"<[email-masked]>
>> Cc:<[email-masked]>

Sorry for mentioning SNMP.  I don't really want to go off on that tangent.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 17 Nov 2011 10:05:51 +0800

>   ability to check at module compile time that there are no name clashes).

I prefer b) since it solves the problems and requires no changes to the specs. I think submodules should be used only within a coordinated group of developers where it does not really limit flexibility. 

Lada

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Nov 2011 18:58:51 -0800

> I prefer b) since it solves the problems and requires no changes to the specs. I think submodules should be used only within a coordinated group of developers where it does not really limit flexibi…

But our import/include by revision is close to worthless, so why should we force
people to use it?  If sec. 10 of RFC 6020 is followed properly, then a client
can use most likely use any newer revision.  Forcing the inclusion of a specific
revision is too fragile.

The python world solved this problem correctly (install_requires)

   install_requires=[
         "TurboGears >= 1.0.4.3",
         "SQLObject>=0.8,<=0.10.0"
     ],

This much more powerful mechanism actually solves the real problems.
Usually, a dependency just needs to be the same or newer than a specific version.
Let's say something in SQLObject is removed from version 0.11.0 that we were using.
For YANG, that would mean a node status was changed to obsolete.
In that case, you want to update the 'rule' to say any module revision from 0.8 through 0.10.0 is OK.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 17 Nov 2011 11:19:23 +0800

> can use most likely use any newer revision.  Forcing the inclusion of a specific

How can a client use any newer revision if an exact revision date is specified? 

> revision is too fragile.

It may be less flexible but I don't get why this is fragile.

>        "SQLObject>=0.8,<=0.10.0"

But these are independent packages with no coordination of versions, so it is IMO an analogy of YANG modules, not submodules. It would be indeed useful to have this order relation for "import" but I don't think it is necessary for "include". I understand submodules as an analogy to e.g. splitting the C source code into multiple files.

Lada  

--
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 16 Nov 2011 20:05:03 -0800

> How can a client use any newer revision if an exact revision date is specified?

Exactly.  It cannot, because of the limitations of YANG.
If the module revision rules are followed, then a non-obsolete definition will be usable
in any newer revision.   The only way to get this robust behavior is to avoid use of
import/include by revision.

> It may be less flexible but I don't get why this is fragile.

Because an update on the server may not impact the client, but the dependency
says it does.  If the server updated module X, it will have to keep supporting
the old and new revisions, when it is sufficient to just advertise the newer revision.
You have to update the module revision just to update the import-by-revision date.
This appears as if the feature changed when it may not have changed at all.

>>         "TurboGears>= 1.0.4.3",

Then IMO we don't have any problem with submodules.
They work as expected.  A client can only manage the life cycle of
the entire module.  If there is unnecessary churn, then too bad.

Our NMS people figured out pretty quickly that one revision for
the entire kitchen sink data model was not what they wanted,
so we changed all the submodules to modules.

Andy