SID Extension to efficiently manipulate YANG Data Models
Laurent Toutain
Institut MINES TELECOM; IMT Atlantique
2 rue de la Chataigneraie
CS 17607
35576 Cesson-Sevigne Cedex
France
Laurent.Toutain@imt-atlantique.fr
Manoj Gudi
Institut MINES TELECOM; IMT Atlantique
2 rue de la Chataigneraie
CS 17607
35576 Cesson-Sevigne Cedex
France
manoj.gudi@imt-atlantique.net
Javier A. Fernandez
Institut MINES TELECOM; IMT Atlantique
2 rue de la Chataigneraie
CS 17607
35576 Cesson-Sevigne Cedex
France
javier-alejandro.fernandez@imt-atlantique.fr
26 March 2024
t2t Research Group
As the Internet of Things (IoT) systems are becoming more pervasive with their rapid adoption, they are also becoming more complex in their architecture. Hence, a tool is required to generate prototype code based on the YANG models for constrained devices [RFC7228] to improve interoperability and increase the reusability of software components. A novel approach is introduced in this document to generate software prototypes (also called stubs) in the C language for the CORECONF protocol [I-D.ietf-core-comi] using YANG Schema Item iDentifiers (YANG SID [I-D.ietf-core-sid]). These stubs greatly reduce the complexity of navigating the CORECONF structure by abstracting the corresponding YANG SIDs. This document elaborates on the procedure to generate YANG SIDs for a given YANG model of a system, which then generates C stubs using the tools developed by the authors with the help of an example (sensor.yang file).
Introduction
The YANG modeling language is very popular for node configuration and retrieving the the state of a device.
XML and JSON are also two popular formats to serialize data in conformance with the data model. Recently, a new serialization format has been published, allowing a more compact representation of the information. This new format is based on CBOR and the YANG identifier; instead of being represented by a unique ASCII string, values are mapped to an unique integer.
The mapping between strings and integer is kept in a .sid file usually generated by tools such as pyang.
This document presents some extensions to the sid files that allow to ease the conversion between CBOR and JSON, as well as manipulating YANG Data Models in a constraint environment.
YANG Data model
YANG is a modeling language to structure information and check its conformity. Each element of a YANG Data Model is identified through an unique identifier. YANG is based on a hierarchical approach. During the serialization phase, data is represented either in XML or JSON. In this document we will use the JSON representation. [RFC7951] indicates how to form a JSON structure conforming to a YANG Data Model:
- Leaves are represented as JSON objects, the key being the leaf name.
- Lists are defined through arrays.
- Identityref is a string containing the identifier in the YANG naming hierarchy for an identity.
The YANG Data Model, in Figure 1, is used to illustrate how data will be serialized. It defines a container representing a physical object able to perform several measurements. The physical object is battery powered, has a status LED able to change color, and a list of attached sensors returning an integer value.

Example of a YANG Data Model for sensors

<CODE BEGINS> file "sensor.yang"
module sensor {
  yang-version 1.1;
  namespace "http://sensorexample.in/sensor-example";
  prefix sen1;

  identity battery-indicator-base-type {
    description
      "a base identity for battery level indicator";
  }

  identity high-level {
    base battery-indicator-base-type;
  }

  identity med-level {
    base battery-indicator-base-type;
  }

  identity low-level {
    base battery-indicator-base-type;
  }

  typedef battery-level {
    type identityref {
      base battery-indicator-base-type;
    }
  }

  container sensorObject {
    leaf statusLED {
      type enumeration {
        enum green  {value 0;}
        enum yellow {value 1;}
        enum red {value 2;}
      }
    }
    leaf battery {
        type battery-level;
    }
    list sensorReadings {
      key "index";
      leaf index {
        type uint8;
      }
      leaf sensorValue {
        type uint32;
      }

    }
  }
}
<CODE ENDS>

The YANG tree regarding this Data Model is given in Figure 2. The tree displays the module hierarchy. For the module "sensor", a container "sensorObject" contains two leaves ("statusLED", "battery") and a list ("sensorReadings").

YANG tree associated to the DM.

$ pyang -f tree sensor.yang
module: sensor
  +--rw sensorObject
     +--rw statusLED?        enumeration
     +--rw battery?          battery-level
     +--rw sensorReadings* [index]
        +--rw index          uint8
        +--rw sensorValue?   uint32

JSON serialization
An example of data, serialized with JSON, and conforming to the YANG Data Model, is given in Figure 3. Embedded JSON Objects allow to represent the hierarchy. The key of the outer JSON Object is composed of the module name and the container name. The embedded JSON Object associated to this contains the leaves as keys. The YANG list is represented by an JSON Array.

JSON structure conform to the YANG DM.

{
  "sensor:sensorObject": {
    "statusLED": "green",
    "battery": "sensor:med-level",
    "sensorReadings": [
      {
        "index": 0,
        "sensorValue": 42
      },
      {
        "index": 1,
        "sensorValue": 22
      }
    ]
  }
}

CBOR Serialization
JSON notation is verbose for constrained networks and objects. To optimize the size of the representation, [RFC9254] defines a CBOR serialization, also used in CORECONF. YANG ASCII identifiers are replaced by unique number. In JSON, the uniqueness is guaranteed by the "namespace" URI, as shown in Figure 1. By construction, the rest of the identifiers are unique.
In CORECONF, the uniqueness is guaranteed through the use of positive integers called SID, which replace the ASCII identifiers. [I-D.ietf-core-sid] defines the allocation process. Module developers may ask for a SID range from their authority. For example, for an IETF module, the IANA will allocate a SID range.
The Figure 4 shows an example of this conversion. The range is arbitrarily fixed between 60000 and 60099. Note that the module, the identities, and the leaves have an assigned SID.

JSON structure conform to the YANG DM.

$ pyang --sid-generate-file=60000:100 --sid-list sensor.yang

SID        Assigned to
​---------  --------------------------------------------------
60000      module sensor
60001      identity battery-indicator-base-type
60002      identity high-level
60003      identity low-level
60004      identity med-level
60005      data /sensor:sensorObject
60006      data /sensor:sensorObject/battery
60007      data /sensor:sensorObject/sensorReadings
60008      data /sensor:sensorObject/sensorReadings/index
60009      data /sensor:sensorObject/sensorReadings/sensorValue
60010      data /sensor:sensorObject/statusLED

File sensor@unknown.sid created
Number of SIDs available : 100
Number of SIDs used : 11

To perform the allocation, the pyang utility generates a .sid file in the JSON format, resulting in a YANG Data Model specified in [I-D.ietf-core-sid]. The Figure 5 gives an excerpt.

JSON structure conform to the YANG DM.

{
  "assignment-range": [
    {
      "entry-point": 60000,
      "size": 100
    }
  ],
  "module-name": "sensor",
  "module-revision": "unknown",
  "item": [
    {
      "namespace": "module",
      "identifier": "sensor",
      "status": "unstable",
      "sid": 60000
    },
    {
      "namespace": "identity",
      "identifier": "battery-indicator-base-type",
      "status": "unstable",
      "sid": 60001
    },
    ...

The serialization in CBOR of the JSON example in Figure 3 is given in Figure 6. Compared to the compacted representation of the JSON structure (152 Bytes), the CORECONF structure is 24 bytes long. Although the size is different, the structure remains the same. It is composed of embedded CBOR Maps (equivalent of JSON Object). The first key is a SID (60005 corresponds to the container). Embedded CBOR Maps use a delta notation to encode the keys. The key 5 corresponds to SID 60005+5, thus to the leaf "statusLED". Key 2 in the second CBOR Map corresponds to the YANG list "sensorReadings", and the elements of the list are stored in a CBOR Array.
Note that in this example, the enum for "statusLED" (60005+5) is an integer and the identityref for "battery" (60005+1) is also an integer pointing to the SID "med-level" (60004).

JSON structure conform to the YANG DM.

b'a119ea65a305000119ea640282a2010002182aa201010216'

Diagnostic notation:
{60005:
  {5: 0,
   1: 60004,
   2: [
     {1: 0,
      2: 42},
     {1: 1,
      2: 22}
     ]
  }
}

Conversion between JSON and CBOR
Even if the conversion between CBOR and JSON formats might look obvious, including data compatible with a YANG Data Model is not so trivial. The reason is that JSON uses ASCII identifiers for readability and CBOR prefers integers for compactness. The Table 1 summarizes some YANG types when coded in JSON ([RFC7951]) or CBOR ([RFC8949]).

YANG basic types in JSON and CBOR.
YANG | JSON | CBOR
int8, int16, int32, uint8, uint16, uint32 | number | +int, -int
int64, uint64 | string | +int, -int
decimal64 | string | CBOR Tag 4
binary | base64 | byte array
bits | string | array
boolean | boolean | boolean
identityref | string | +int
enumeration | string | +int
The conversion from CBOR to JSON is not direct, since the YANG type needs to be considered. For instance, a integer could be converted into a number, an enum, or an identityref. Note that for union, the conversion is simple since some CBOR Tags may be used to indicate how the integer is converted, but outside of the union, for a single value, there is no such clue.
On the other direction, a JSON string may correspond to a 64-bit long number, an enumeration, or an identityref.
To perform the conversion, the YANG type is needed, but popular tools such as yanglint or pyang are not currently manipulating CBOR representation. Furthermore, these tools cannot run on constrained devices. To overcome these problems, we propose to extend the .sid file with more information. The modified pyang code (see https://github.com/ltn22/pyang/tree/sid-extension) adds useful information to the .sid file when the "--sid-extension" argument is provided.
This extension contains a "type" key in the JSON Object describing the mapping between the SID and the identifier if the node is a leaf (see Figure 7).

SID "type

...
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject",
      "status": "unstable",
      "sid": 60005
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/battery",
      "status": "unstable",
      "sid": 60006,
      "type": "identityref"
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/sensorReadings",
      "status": "unstable",
      "sid": 60007
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/sensorReadings/index",
      "status": "unstable",
      "sid": 60008,
      "type": "uint8"
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/sensorReadings/sensorValue",
      "status": "unstable",
      "sid": 60009,
      "type": "uint32"
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/statusLED",
      "status": "unstable",
      "sid": 60010,
      "type": {
        "0": "green",
        "1": "yellow",
        "2": "red"
      }
    }
...

The "type" key added to leaf nodes contains several information:
- when the "type" key is not present, the node is a not leaf and there is no need for type conversion.
- when the value of "type" is a string, it gives the YANG type of the leaf, as defined in the YANG Data Model. If the YANG type derives from an identityref, the value "identityref" is given instead of the YANG type specified in the module. In that case, in the CBOR notation, there is a pair "{ ...., SID1: value1, ...} in the CBOR Map. A first lookup at the .sid file for SID1 returns that the type is "identityref", a second lookup for value1 in the "identity" namespace, returns the ASCII identifier.
- when the value of "type" is a JSON Object, then the leaf is a YANG enumeration, and the CBOR Map gives the mapping between integer and strings.
- when the value of "type" is a JSON Array, then the YANG leaf is a union. Elements of the Array list the possible types. In that case, the conversion is leaded by the CBOR Tags associated to the value.
We developed the pycoreconf Python module to facilitate the conversion between JSON and CBOR (https://github.com/ltn22/pycoreconf). The Figure 8 gives an example of a Python script using this module. It takes as input the .sid file and a JSON structure. It transforms it into a CBOR structure, and back to JSON representation.

pycoreconf module.

# pycoreconf sample: "basic"
# This script demonstrates the basic usage of pycoreconf
#  using a simple YANG datamodel.

import pycoreconf
import binascii
import cbor2 as cbor

# Create the model object
ccm = pycoreconf.CORECONFModel("sensor@unknown.sid")

# Read JSON configuration file
config_file = "output.json"

with open(config_file, "r") as f:
    json_data = f.read()
print("Input JSON config data =\n", json_data, sep='')

# Convert configuration to CORECONF/CBOR
cbor_data = ccm.toCORECONF(config_file) # can also take json_data
print("Encoded CBOR data (CORECONF payload) =", binascii.hexlify(cbor_data))
print (cbor.loads(cbor_data))

# Decode CBOR data back to JSON configuration data
decoded_json = ccm.toJSON(cbor_data)
print("Decoded config data =", decoded_json)

The resulting JSON structure of Figure 3 is given in Figure 9. It shows that the JSON format can be converted to CBOR and vice versa, just by using the extended .sid file.

pycoreconf module.

Input JSON config data =
{
  "sensor:sensorObject": {
    "statusLED": "green",
    "battery": "sensor:med-level",
    "sensorReadings": [
      {
        "index": 0,
        "sensorValue": 42
      },
      {
        "index": 1,
        "sensorValue": 22
      }
    ]
  }
}

Encoded CBOR data (CORECONF payload) = b'a119ea65a305000119ea640282a2010002182aa201010216'
{60005: {5: 0, 1: 60004, 2: [{1: 0, 2: 42}, {1: 1, 2: 22}]}}
Decoded config data = {"sensor:sensorObject": {"statusLED": "green", "battery": "med-level", "sensorReadings": [{"index": 0, "sensorValue": 42}, {"index": 1, "sensorValue": 22}]}}

Navigating the CORECONF structure.
As we saw, the CORECONF data is structured as a tree. The Figure 10 gives an example for the example module described in Figure 1. The numbers are the SID associated to the YANG Data Model. [I-D.ietf-core-comi] defines how queries are made on this structure to get the full tree or a sub tree.

pycoreconf module.

                      sensorObject
                         60005
           battery     LED |    sensorReadings
                +----------+----------+
                |          |          |
              60006      60010      60007
                                      |
               +-----------+----------+-----------+
        index  |           |          |           |  sensorValue
        +---------+ +----------+ +----------+ +----------+
        |         | |          | |          | |          |
        60008 60009 60008  60009 60008  60009 60008  60009

Requesting the SID corresponding to the module (60005), will return the full tree. Requesting "battery" leaf (60006) will return the leaf and the associated value, while requesting "sensorReadings" (60007) will return the subtree with all its values. Going deeper in this sub-tree imposes the use of keys. For instance, to get a specific "sensorValue" one must provide the key "index" (60008).
In [I-D.ietf-core-comi], the series of keys needed to reach a specific element is provided either in the query string or in the fetch. Only one list is provided in Figure 10, but several lists can be embedded, and each level may require one or more keys. So, to be processed, the YANG Data Model as to be known.
To avoid parsing the YANG Data Model on a constrained device, we propose to summarize this information into a JSON Object, generated by pyang with the "--sid-extension" argument.

pycoreconf module.

  "key-mapping": {
    "60007" :
        [60008]
  }

The "key-mapping" key points out to a JSON Object, where the JSON key is a SID corresponding to a YANG list key, and the value is a JSON array containing the SIDs acting as a YANG key for that YANG list. The "key-mapping" JSON Object indicates:
- that a SID is a YANG list, since its value appears as a JSON key in the "key-mapping" structure.
- the number of elements in the corresponding array indicates how many items are involved as a YANG list key. The value for these items can be taken from the request.
- the array also indicates which SIDs contain the value that must be compared to the YANG list key sent in the request.
With the information contained in the "key-mapping" structure, it is possible to browse any CORECONF structure and return the appropriate requested SID.
Linking to real values
In some YANG Data Models, as the one provided in Figure 1, some leaf nodes might encompass information that extends beyond the model's boundaries. For example, the "statusLED" leaf may require an action to be triggered when a value is written onto it, or the managed device might need to interact with a physical sensor when the "sensorValue" leaf is queried. On the other hand, certain leaf nodes, such as "index", do not require any interaction as their values are directly stored in the database.
We wrote a script taking the .sid file obtained from pyang with the "--sid-extension" argument to produce template C code for these functions. We try keep SID transparent and use the YANG identifiers as much as possible, which are easier to manipulate by a programmer.
The tool aims to generate aliases for identifiers that also include details about their associated SIDs. This eliminates the need for developers to recall or verify SIDs during software development.
- The tool is hosted at https://github.com/manojgudi/ccoreconf/tree/stub_generation/tools
- For the SID file above (sensor@unknown.sid), the second argument in the command- "sensor_prototypes" instructs the tool to create two files sensor_prototypes.h and sensor_prototypes.c for writing headers and function code, respectively. To generate the C code, simply run the tool:

$ python generateStubs.py sensor@unknown.sid "sensor_prototypes"

//Headers
//-----------
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <cbor.h>

#define  SID_BATTERY             60006
#define  SID_INDEX               60008
#define  SID_SENSORVALUE         60009
#define  SID_STATUSLED           60010

#define read_sensorObject        read_60005
#define read_battery             read_60006
#define read_sensorReadings      read_60007
#define read_sensorValue         read_60009
#define read_statusLED           read_60010

char* keyMapping = "\xa1\x19\xeag\x81\x19\xeah";
enum StatusledEnum {green = 0, yellow = 1, red = 2};

void  read_sensorObject(void);
char *  read_battery(void);
void  read_sensorReadings(uint8_t index);
uint32_t  read_sensorValue(uint8_t index);
enum StatusledEnum read_statusLED(void);

//Code File
//-----------
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include "sensor_prototypes.h"

/*
    This is an autogenerated function associated to
    SID: 60005
    Module: data
    Identifier: /sensor:sensorObject
    function params:
    Stable: false
*/
void  read_sensorObject(void){
    // Initialize the leaf if it has a return type with a default value;
    // Do something with the leaf
}

/*
    This is an autogenerated function associated to
    SID: 60006
    Module: data
    Identifier: /sensor:sensorObject/battery
    function params:
    Stable: false
*/
char *  read_battery(void){
    // Initialize the leaf if it has a return type with a default value;
    char * batteryInstance  = NULL;

    // Do something with the leaf
    // Return the leaf
    return batteryInstance;
}

/*
    This is an autogenerated function associated to
    SID: 60007
    Module: data
    Identifier: /sensor:sensorObject/sensorReadings
    function params: /sensor:sensorObject/sensorReadings/index
    Stable: false
*/
void  read_sensorReadings(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    // Do something with the leaf
}

/*
    This is an autogenerated function associated to
    SID: 60009
    Module: data
    Identifier: /sensor:sensorObject/sensorReadings/sensorValue
    function params: /sensor:sensorObject/sensorReadings/index
    Stable: false
*/
uint32_t  read_sensorValue(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    uint32_t sensorValueInstance  = 0;

    // Do something with the leaf
    // Return the leaf
    return sensorValueInstance;
}

/*
    This is an autogenerated function associated to
    SID: 60010
    Module: data
    Identifier: /sensor:sensorObject/statusLED
    function params:
    Stable: false
*/
enum StatusledEnum read_statusLED(void){
    // Initialize the leaf if it has a return type with a default value;
    enum StatusledEnum statusLEDInstance;

    // Do something with the leaf
    // Return the leaf
    return statusLEDInstance;
}

Understanding Stubs
Let us take a quick look at the generated code snippet to understand how they help abstract SIDs from the developers:
The first part of the sensor_prototypes.h contains pre-processor directives which basically store the identifiers with their corresponding SID values for quick access. The second part contains function prototypes which can be implemented later. Note that there will not be any function prototype generated to read SID keys (in this case index, with SID 60008).

// Aliases of the leaves mapped to their corresponding SID
#define  SID_BATTERY             60006
#define  SID_INDEX               60008
#define  SID_SENSORVALUE         60009
#define  SID_STATUSLED           60010

// Aliases of function prototypes linked to their function name containing SID value for quick and easy access
#define read_sensorObject        read_60005
#define read_battery             read_60006
#define read_sensorReadings      read_60007
#define read_sensorValue         read_60009
#define read_statusLED           read_60010

// The relation between certain identifiers and their sid keys is stored in keyMapping as a CBOR map
char* keyMapping = "\xa1\x19\xeag\x81\x19\xeah";
enum StatusledEnum {green = 0, yellow = 1, red = 2};

// Function prototypes of the getters which can be implemented by the developer later in its corresponding C file
// The return types of the functions are mapped according to the type of the leaf. The getters for non-leaves are mapped as void
void  read_sensorObject(void);
char *  read_battery(void);
void  read_sensorReadings(uint8_t index);
uint32_t  read_sensorValue(uint8_t index);
enum StatusledEnum read_statusLED(void);

Now let us understand what is generated in sid_prototypes.c by looking at the auto-generated function prototype read_sensorValue. The program auto-infers that the leaf sensorValue can only be reached through a valid SID key (that is, index), so a function trying to read sensorValue from its CORECONF instance will require a parameter index. Also, since sensorValue has a well-defined type in C, the function will try to return that uint32_t.
The program also maps the enum type from SID to a corresponding enum type in C, as shown for the leaf statusLED. The value of these enums is as specified in the corresponding YANG model. If unspecified, the values are auto-assigned.
For rest of the non-leaf functions and leaf nodes with type identityref, the program infers it to be a char * (since the types are non-standard).

uint32_t  read_sensorValue(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    uint32_t sensorValueInstance  = 0;

    // Do something with the leaf
    // Return the leaf
    return sensorValueInstance;
}

Normative References
[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels
S. Bradner
March 1997
BCP 14
RFC 2119
DOI 10.17487/RFC2119
[RFC7228]

Terminology for Constrained-Node Networks
C. Bormann
M. Ersue
A. Keranen
May 2014
RFC 7228
DOI 10.17487/RFC7228
[RFC7951]

JSON Encoding of Data Modeled with YANG
L. Lhotka
August 2016
RFC 7951
DOI 10.17487/RFC7951
[RFC8949]

Concise Binary Object Representation (CBOR)
C. Bormann
P. Hoffman
December 2020
STD 94
RFC 8949
DOI 10.17487/RFC8949
[RFC9254]

Encoding of Data Modeled with YANG in the Concise Binary Object Representation (CBOR)
M. Veillette, Ed.
I. Petrov, Ed.
A. Pelov
C. Bormann
M. Richardson
July 2022
RFC 9254
DOI 10.17487/RFC9254
[I-D.ietf-core-sid]

YANG Schema Item iDentifier (YANG SID)
Michel Veillette
Trilliant Networks Inc.
Alexander Pelov
IMT Atlantique
Ivaylo Petrov
Google Switzerland GmbH
Carsten Bormann
Universität Bremen TZI
Michael Richardson
Sandelman Software Works
22 December 2023
Internet-Draft draft-ietf-core-sid-24
[I-D.ietf-core-comi]

CoAP Management Interface (CORECONF)
Michel Veillette
Trilliant Networks Inc.
Peter Van der Stok
consultant
Alexander Pelov
IMT Atlantique
Andy Bierman
YumaWorks
Carsten Bormann
Universität Bremen TZI
4 March 2024
Internet-Draft draft-ietf-core-comi-17