Subject: Re: [netmod] smi2yang-14: unflatten scalars [open]

From: David Spakes <[email-masked]>
Date: Mon, 14 Nov 2011 19:50:51 -0500 (EST)

On Mon, 14 Nov 2011, Juergen Schoenwaelder wrote:

> Subject: Re: [netmod] Last Call: draft-ietf-netmod-smi-yang-01 (20110704)
...
> > Section 8.1
> >
> >    "All leaf statements for scalar objects are created in the top-level
> >    container"
> >
> >   OK -- but I just want to point out that this leads to a model where
> >   you need to be careful with using filters if you don't want to get
> >   all data for a module.
>
> yep - created an open issue for it.

On Mon, 14 Nov 2011, Juergen Schoenwaelder wrote:

> * smi2yang-14: unflatten scalars [open]
>
>   With the resolution of #01, we ended with having all scalars
>   registered right below the top-level node. This makes retrieving
>   just those scalars (often collections of counters) a bit tricky
>   because the naive approach will also retrieve the tables.
>
> ** Solution #14-01
>
>    Ignore this and assume people write proper filters.
>
> ** Solution #14-02
>
>    Move all scalars into an artificially created container, so we end
>    up with:
>
>    foo-mib
>      +-- scalars
>      |     +-- fooScalar1
>      |     +-- fooScalar2
>      +-- fooTable1
>      +-- fooTable2

I _strongly_ dislike Solution #14-02 for SMIv2 documents that group
scalar objects together in logical collections by placing them into
separate branches of the MIB.

I would like to propose another solution in which logical collections
are preserved by having a separate top-level container for each of those
collections.

   foo-fum-mib
     +-- fooScalars
     |     +-- fooScalar1
     |     +-- fooScalar2
     +-- fumScalars
     |     +-- fumScalar1
     |     +-- fumScalar2
     +-- fooTable
     +-- fumTable

The name for each of these top-level containers comes from the parent
of the scalar objects in the collection.  For example: sysDescr,
sysObjectID, sysUpTime, sysContact, sysName, sysLocation, and
sysServices shared a common parent, system (RFC 1213); so the name of
the container for this logical collection of objects would be system.

I understand the difficulty of producing containers in the general case
since multiple descriptors may refer to the same OID value (issue #01).
I would argue, however, that we can treat the parent node of a logical
collection of scalar objects in SMIv2 documents in a way different from
the general case, because it would be rare that two descriptors would
refer to this particular class of node.

Here is the full example for RFC 1213 scalars:

   RFC1213-MIB
     +-- system
     |     +-- sysDescr
     |     +-- sysObjectID
     |     +-- sysUpTime
     |     +-- sysContact
     |     +-- sysName
     |     +-- sysLocation
     |     +-- sysServices
     +-- interfaces
     |     +-- ifNumber
     +-- ip
     |     +-- ipForwarding
     |     +-- ipDefaultTTL
     |     +-- ipInReceives
     |     +-- ipInHdrErrors
     |     +-- ipInAddrErrors
     |     +-- ipForwDatagrams
     |     +-- ipInUnknownProtos
     |     +-- ipInDiscards
     |     +-- ipInDelivers
     |     +-- ipOutRequests
     |     +-- ipOutDiscards
     |     +-- ipOutNoRoutes
     |     +-- ipReasmTimeout
     |     +-- ipReasmReqds
     |     +-- ipReasmOKs
     |     +-- ipReasmFails
     |     +-- ipFragOKs
     |     +-- ipFragFails
     |     +-- ipFragCreates
     |     +-- ipRoutingDiscards
     +-- icmp
     |     +-- icmpInMsgs
     |     +-- icmpInErrors
     |     +-- icmpInDestUnreachs
     |     +-- icmpInTimeExcds
     |     +-- icmpInParmProbs
     |     +-- icmpInSrcQuenchs
     |     +-- icmpInRedirects
     |     +-- icmpInEchos
     |     +-- icmpInEchoReps
     |     +-- icmpInTimestamps
     |     +-- icmpInTimestampReps
     |     +-- icmpInAddrMasks
     |     +-- icmpInAddrMaskReps
     |     +-- icmpOutMsgs
     |     +-- icmpOutErrors
     |     +-- icmpOutDestUnreachs
     |     +-- icmpOutTimeExcds
     |     +-- icmpOutParmProbs
     |     +-- icmpOutSrcQuenchs
     |     +-- icmpOutRedirects
     |     +-- icmpOutEchos
     |     +-- icmpOutEchoReps
     |     +-- icmpOutTimestamps
     |     +-- icmpOutTimestampReps
     |     +-- icmpOutAddrMasks
     |     +-- icmpOutAddrMaskReps
     +-- tcp
     |     +-- tcpRtoAlgorithm
     |     +-- tcpRtoMin
     |     +-- tcpRtoMax
     |     +-- tcpMaxConn
     |     +-- tcpActiveOpens
     |     +-- tcpPassiveOpens
     |     +-- tcpAttemptFails
     |     +-- tcpEstabResets
     |     +-- tcpCurrEstab
     |     +-- tcpInSegs
     |     +-- tcpOutSegs
     |     +-- tcpRetransSegs
     |     +-- tcpInErrs
     |     +-- tcpOutRsts
     +-- udp
     |     +-- udpInDatagrams
     |     +-- udpNoPorts
     |     +-- udpInErrors
     |     +-- udpOutDatagrams
     +-- egp
     |     +-- egpInMsgs
     |     +-- egpInErrors
     |     +-- egpOutMsgs
     |     +-- egpOutErrors
     |     +-- egpNeighIntervalHello
     |     +-- egpNeighIntervalPoll
     |     +-- egpNeighMode
     |     +-- egpNeighEventTrigger
     |     +-- egpAs
     +-- ifTable
     ...

I would hate to see any SMIv2 document that contains a lot of scalar
objects divided into logical collections be converted in such a way
that the collections are lost.

With equal emphasis, I don't like the idea that other levels of
containment are created that don't exist in the original SMIv2
document.  For one reason, this implies a logical collection that
isn't explicitly specified by the original SMIv2 document.  (If my
suggested solution is not accepted by the working group, then I
would prefer Solution #14-01 to any other solution.)

Note that if an SMIv2 document defines scalar objects at a layer that
is immediately below the MODULE-IDENTITY macro, then the container
for this logical collection would have the same name as the
MODULE-IDENTITY.  This would resemble one of the previous iterations
of Solution #14-02 for some SMIv2 documents.

Note also that if an SMIv1 document defines scalar objects at the
top layer of the module, or if an SMIv2 document defines scalar
objects at the top layer of the module (not below the MODULE-IDENTITY),
then these objects would not be inside a container in the resulting
YANG document.  They would exist at the top layer of the YANG document.
This resembles Solution #14-01 but only because the converted document
retains the absense of a logical collection for these objects, same as
in the original document.

-dss

-------------------------------------------------------------
 David Spakes                       email:   [email-masked]
 SNMP Research                      voice:   +1 865 573 1434
 3001 Kimberlin Heights Road          fax:   +1 865 573 9197
 Knoxville, TN  37920-9716  USA      http://www.snmp.com
-------------------------------------------------------------

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 15 Nov 2011 03:31:38 +0100

> the container for this logical collection of objects would be system.

A clarification question. If I have

foo OBJECT IDENTIFIER ::= { here 1 }
bar OBJECT IDENTIFIER ::= { here 1 }

and

a OBJECT-TYPE --...--- ::= { foo 1 }
b OBJECT-TYPE --...--- ::= { bar 1 }

then I would get

foo/a
bar/b

and we would make the assumption that { foo 1 } would never change to
{ bar 1 }, although I would believe this is a legal change according
to the SMIv2 rules.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 15 Nov 2011 09:25:17 +0100 (CET)

> are preserved by having a separate top-level container for each of those

I like this solution.  The logical groupings from the MIBs are kept;
the structure gets better and it helps during retrieval (filtering).

However, there are some corner cases that we have to handle (even if
"handle" means document bad behaviour).

For example, what would the result of this be:

  fooScalar OBJECT-TYPE 
      ....
      ::= { foo 1 2 }

?

And what happens if, in a new revision, this is changed to:

  bar OBJECT IDENTIFIER ::= { foo 1 }

  fooScalar OBJECT-TYPE 
      ....
      ::= { bar 2 }

Would you limit this logical grouping to scalars only, or also put
tables there?

/martin

---

From: David Spakes <[email-masked]>
Date: Tue, 15 Nov 2011 15:12:21 -0500 (EST)

> Would you limit this logical grouping to scalars only, or also put

Martin,

I would keep the tables at the top level.  The result is elegant.

Everything that appears at the top level except notifications
represents a logical collection of objects that is reflected in
the object identifiers.

1. container with scalars (a logical collection of scalar objects)

    The object identifier for each scalar object is equal to the
    smiv2:oid of the container + "." + smiv2:subid of the object.

2. container with a list (a logical collection of columnar objects)

    The object identifier for the list is equal to the
    smiv2:oid of the container + "." + smiv2:subid of the list.

    The object identifier for each columnar object is equal to the
    smiv2:oid of the container + "." + smiv2:subid of the list
    + "." + smiv2:subid of the object.

3. augment (a logical collection of columnar objects)

    The object identifier for each columnar object is equal to the
    smiv2:oid of the augment + "." + smiv2:subid of the object.

Each notification also has a logical collection of objects, but
this collection is not reflected in the object identifiers.
Therefore, the object identifier for each object in the collection
(whether scalar or tabular) must be specified explicitly using
smiv2:oid.

-dss

---

From: David Spakes <[email-masked]>
Date: Tue, 15 Nov 2011 15:42:47 -0500 (EST)

> represents a logical collection of objects that is reflected in

As soon as I sent the e-mail, I thought of one exception to my
statement above.  A table can have an external index, and the
object identifier does not reflect its inclusion in the logical
collection.

For example, in module RMON2-MIB, the list serialConfigEntry has
the external index ifIndex, and the object identifier would
need to be specified using an smiv2:oid statement...

  container serialConfigTable {
    config false;
    description
       "A table of serial interface configuration entries.  This data
        will be stored in non-volatile memory and preserved across
        probe resets or power loss.

        This table has been deprecated, as it has not had enough
        independent implementations to demonstrate interoperability to
        meet the requirements of a Draft Standard.";
    smiv2:oid "1.3.6.1.2.1.16.19.10";

    list serialConfigEntry {
      key "ifIndex";
      status deprecated;
      description
       "A set of configuration parameters for a particular
        serial interface on this device.  If the device has no serial
        interfaces, this table is empty.

        The index is composed of the ifIndex assigned to this serial
        line interface.";
      smiv2:subid 1;

      leaf ifIndex {
        type leafref {
          path "/if-mib:ifTable/if-mib:ifEntry/if-mib:ifIndex";
        }
        smiv2:oid "1.3.6.1.2.1.2.2.1.1";
      }
      ...
    }
  }
  ...

So I must revise the usage rules as follows:

  extension oid

    1) MUST be used in the first level of containment within the module.
       For example, in augments, top-level containers, notifications,
       and alias.

    2) SHOULD NOT be used below the first level of containment within a
       module, except in the following cases:

         a) to specify the object identifier for key objects of type
            leafref defined within lists

         b) to specify the object identifier for leaf objects defined
            within notifications

         c) as a substatement to an alias

  extension subid

    1) MUST NOT be used above the second level of containment within a
       module.

    2) MUST be used at every level below the first level of containment
       within a module to determine the object identifier for every
       scalar object and columnar object translated from the SMIv2
       module, not including objects defined within a notification
       and key objects of type leafref defined within a list.

-dss

---

From: David Spakes <[email-masked]>
Date: Thu, 17 Nov 2011 11:44:33 -0500 (EST)

On Wed, 9 Nov 2011, Juergen Schoenwaelder wrote:

> ... If you need to
> translate them, simply fix the module before running the translator.
> The IETF MIB modules that have such problems and are shipped with
> smidump have been fixed manually.

Martin,

You asked me three questions on Tuesday.  I answered one right away
(not quoted above) but took time to ponder the other two (quoted above)
before answering.

I would have the ietf-yang-smiv2 document specifically mention the
above two cases.

  1. Let the spec say that an SMIv2 document that defines the OID
     of an object with an expression like "foo 1 2" can not be
     translated to YANG unless "foo 1" is resolved elsewhere in the
     module (or perhaps by an imported module).

  2. If there is a later revision of the same module that resolves
     "foo 1", then the spec should state as a Best Current Practice
     what Juergen said earlier: simply fix the module before running
     the translator.

Personally, I have seen this "foo 1 2"-type expression used in some
SMIv2 documents to identify the series of branches leading down to
the MODULE-IDENTITY.  But I can't specifically recall a single time
seeing it used below the MODULE-IDENTITY.  That's the only place
where it would matter.

-dss

---

From: David Reid <[email-masked]>
Date: Thu, 17 Nov 2011 12:03:06 -0500

>      module (or perhaps by an imported module).

What if just name the container foo.1 ('.' is legal in a yang identifier).

>      what Juergen said earlier: simply fix the module before running

The revised module would have a new container named bar and lose the
container named foo.1. That's bad. But this seems unlikely to happen
in practice.

-David Reid

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 22 Nov 2011 23:21:54 +0100

On Thu, Nov 17, 2011 at 12:03:06PM -0500, David Reid wrote:

[...]

I am having some hard time to follow the discussion. It seems we mix
up issues - the way we write down OIDs really has nothing to do with
the question how many containers we generate and how we name them. And
from previous discussions, we know that finding names is brittle since
legal SMIv2 updates can cause incompatible results. This is why we
dropped all the containers several months ago - for the sake of
robustness.

To resolve ambiguities (if a parent of object foo has two
descriptors), I thought looking at the definition of foo might be a
good heuristic. But, it is still a heuristic. And it turns out
implementing it may not be that easy - I looked at the libsmi code and
it does not maintain knowledge how an OID was defined. So to implement
this solution, the frontend parser needs to be extended... This is not
impossible to do but raises the bar quite a bit.

While it is true that OID value expressions for OBJECT-TYPES that are
not of the form { <descriptor> <number> } are rare, other forms do for
sure exist for notifications and MODULE-IDENTITY OIDs, even in IETF
modules.  I guess I am not so much concerned about IETF modules but
more about the large number of enterprise modules - some of them I
know look really creative.

So where are we with this discussion? Is this a fair summary?

1) The toplevel container is named after the module.

2) All tables are directly below the toplevel container.

3) Augments must anyway be below the toplevel container, says YANG.

4) Scalars are moved into containers that are their parent. The name
   of that parent is identified in the following way:

   a) If the parent in the OID tree of the object has only a single
      descriptor, this name is used.

   b) Otherwise, if the OID of the object is defined in the form {
      <descriptor> <number> } or { <something> <descriptor>(<number>)
      <number> }, then the parent is called <descriptor>. We assume
      that people do not make random changes while revising MIB
      modules - if they do we fail producing always the same YANG
      translation.

   c) Otherwise, if the parent does not have a descriptor or there are
      multiple descriptors left to choose from, the translation is
      aborted with a suitable explanation.

/js

---

From: David Spakes <[email-masked]>
Date: Tue, 22 Nov 2011 17:59:02 -0500 (EST)

On Tue, 22 Nov 2011, Juergen Schoenwaelder wrote:

> sure exist for notifications

I don't think this issue applies to notifications.  We aren't
trying to preserve logical collections of notifications by putting
them inside a container.  As I understood, the smi2yang process as
it stands now would result in notifications all being at the top layer.
The objects delivered by a notification would be contained by the
notification.

> and MODULE-IDENTITY OIDs,

I'm not sure how this applies to smi2yang-14.  It is possible that
the immediate parent of some scalar objects could be the MODULE-IDENTITY,
but, and you do have the name of the MODULE-IDENTITY which can be used
as the name of the container for those scalars.

I can't think of a case where the translator would need to know the
name of the parent of a MODULE-IDENTITY.

> even in IETF

I saw that discussed in the minutes, etc.  If I had been there, I would
have been in agreement.

> 2) All tables are directly below the toplevel container.

The last sample translation I saw put the list inside a container; e.g.,

  container ifTable {
    list ifEntry {
    }
  }

> 3) Augments must anyway be below the toplevel container, says YANG.

Yes, and the name of the table and entry are only preserved through
an alias statement; i.e.,

  augment "path" {
    leaf ifName {
    }
    ...
    leaf Alias {
    }
  }
  alias ifXTable { ... }
  alias ifXEntry { ... }

>       descriptor, this name is used.

Yes; e.g.,

  container system {
     leaf sysDescr {
     }
     ...
     leaf sysServices {
     }
  }

>       modules - if they do we fail producing always the same YANG

The minutes weren't very clear to me if this was discussed at the
meeting.

If you have <descriptor>.<number>.<object> and try to collapse
this into

  container <descriptor> {
     leaf <object> {
     }
  }

then you may be losing some logical collections in translation if
there is more than one <number>.

David Reid suggested

  container <descriptor>_<number> {
     leaf <object> {
     }
  }

but the danger there is different revisions of the document would likely
have the leaf objects in different containers.

IMHO, we should drop b) from consideration and go straigt from a) to c).

>       aborted with a suitable explanation.

Yes.

-dss

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 23 Nov 2011 00:17:58 +0100

> I saw that discussed in the minutes, etc.  If I had been there, I would

Good. 
 
> The last sample translation I saw put the list inside a container; e.g.,

Thats correct - the container represents the table, the list
represents the 'row'. You agree to keep it this way?

Yes.

> >       descriptor, this name is used.

If system is the only name known for sysDescr etc.
 
> there is more than one <number>.

No, the situation here is you have

a OBJECT IDENTIFIER ::= { somewhere 1 }
b OBJECT IDENTIFIER ::= { somewhere 1 }

foo OBJECT-TYPE --...-- ::= { a 1 }

and you choose 'a' as the name of the parent. Similarily, if you have

bar OBJECT-TYPE --...-- ::= { somewhere c(1) 2 }

you put bar under 'c'.
 
> have the leaf objects in different containers.

This would have been case c) in my description.

> >       aborted with a suitable explanation.

Anyway, if people arbitrarily rename OID descriptors, then we lost 
because we can't detect this. That is, if someone changes

a OBJECT IDENTIFIER ::= { somewhere 1 }
foo OBJECT-TYPE --...-- ::= { a 1 }

to

b OBJECT IDENTIFIER ::= { somewhere 1 }
foo OBJECT-TYPE --...-- ::= { b 1 }

then we will create different YANG modules. Such changes sometimes
happen if company a buys company b and someone is tasked to apply the
name change blindly. But then likely they also rename the module, in
which case we are fine again - kind of. I guess we just document these
limitations / assumptions we make.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 23 Nov 2011 09:29:22 +0100 (CET)

> foo OBJECT-TYPE --...-- ::= { b 1 }

But this is illegal in SMIv2:

   Note that changing the descriptor associated with an existing object
   is considered a semantic change, as these strings may be used in an
   IMPORTS statement.

(this is also mentioned in 3.6 of 2578)

and we don't have to cover illegal changes.

[Didn't we discuss this before...?  I forgot the outcome...]

However, going from

to

is legal, and in this case the YANG will be different.

But IMO, documenting this is good enough.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 23 Nov 2011 11:32:23 +0100

> [Didn't we discuss this before...?  I forgot the outcome...]

If RFC 2578 would have been that clear. The question is whether an
ASN.1 value assignment is an "object". Unfortunately, RFC 2578 does
not say "anything that might be used in an IMPORT must never change"
even though that is technically what is needed. But yes, this is a
corner case we can likely ignore.
 
> But IMO, documenting this is good enough.

OK

/js

---

From: David Spakes <[email-masked]>
Date: Wed, 23 Nov 2011 15:42:27 -0500 (EST)

> represents the 'row'. You agree to keep it this way?

Yes, that seems fine.

-dss