Subject: [core] option numbers greater than 65535

From: "Peter A. Bigot" <[email-masked]>
Date: Mon, 21 Oct 2013 07:01:11 -0500

Nothing in coap-18 that I can see prevents a message from carrying an
option with a number greater than 65535, though options above 65804 may
need to be reached in steps.  This was observed by Mykyta Yevstifeyev in
message 1521 (which also cites a relevant MUST requirement in RFC5226),
but at a time when fenceposting still existed and there was a more
effective practical upper bound.

Did I miss something?  If not, is this worthy of a clarification or
erratum?  I believe some current implementations will mask off higher
bits in this case, so option number 65550 would be misinterpreted as a
Max-Age option.

I'd rather not add support for higher numbers into CoAPy, but if the
specification doesn't prohibit them I'll need to do that.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 21 Oct 2013 14:52:40 +0200

On Oct 21, 2013, at 14:01, "Peter A. Bigot" <[email-masked]> wrote:

> need to be reached in steps.  

Right, the delta encoding does not (cannot) prevent expressing numbers > 16 bit.

The CoAP Option Number Registry only provides policy for option numbers ≤ 65535.

So there is no good way to use an option number ≥ 2**16.

I'd say you can use 16-bit unsigneds to represent option numbers inside an implementation, 
and you can happily 4.02 (Bad Option) on any message containing one that decodes to ≥ 2**16.
(Technically, you could keep and interpret the low order bits to distinguish critical from elective etc., but I don't see a good reason why.)

> I believe some current implementations will mask off higher

My advice to a receiver would be not to do that.
My advice to a sender would be not to rely on the receivers' correctness in this case (but then I wouldn't know to rely on that for what purpose...).

Matthias: is there a good place in draft-kovatsch-lwig-coap where we can discuss these things?
Or should this collected wisdom go into the core roadmap (RFC 4815 style)?

Grüße, Carsten

---

From: "Peter A. Bigot" <[email-masked]>
Date: Mon, 21 Oct 2013 08:49:18 -0500

> (Technically, you could keep and interpret the low order bits to distinguish critical from elective etc., but I don't see a good reason why.)

I don't like this approach for a couple reasons.

First, a specification should clearly answer any implementer's "What if
I do this?" questions without requiring a search for intent in the
working group's list archives or any draft or even published "best
practices" documents.  If this can't be resolved in coap-xx before it
becomes an RFC, it needs to be an erratum which can be found by direct
link from the IETF RFC pages.

Second, using 4.02 unnecessarily crosses a layer boundary.  That a "big
option" (one with a number above 65535) is even expressible is due to a
weakness in message encoding, and it should be diagnosed at the message
(CON/NON/ACK/RST) layer not at the REST transaction (request/response)
layer, just as messages with unrecognized critical options are rejected.

What I propose is that the following text resolve this:

    The Option Number MUST be a value in the range 0..65535, although the
    delta encoding permits representations of larger numbers.  Endpoints
    MUST NOT send messages with larger option numbers.  Endpoints MUST
    reject any message with a larger option number.

That probably belongs somewhere around 3.1, but maybe 4.1 or 12.2.  The
last MUST could be a SHOULD, if the IETF model of "undefined behavior"
is like the C language: that the implementation can do whatever it wants
(including masking off high bits of option numbers) if it's fed
something that it was not supposed to be fed.

This text should clearly convey to the implementer the expectations of
the protocol.  If it doesn't, let's refine it.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 21 Oct 2013 16:32:26 +0200

> practices" documents.  

Yes.  It also should be less than a thousand pages.
More important, it needs to be *done* at some point.
We reached that point with the core spec 2013-07-15.

Data point: People have been able to build the web with a document as precise as RFC 2616.
Only now, 15 years later, do we pick up the loose ends and try to write up httpbis (approximately doubling the number of pages).

I'm not saying having a perfect specification would be undesirable, it's just that perfection here is empirically uncorrelated with the quality of the interoperability.

> If this can't be resolved in coap-xx before it

This point is unlikely to ever cause interoperability issues, so it doesn't seem to have very high priority.

For documenting things like this, I think that RFC 4815 style documents work better than lists of errata.  Let's discuss in Vancouver how we want to handle this phase.
(As another data point, HTTP went from 2068 to 2616 within 2.5 years.  We could do a respin of the spec after a couple of years of experience, too.  At a cost of several hundred thousands of dollars of engineer time, so that respin should be worth it.)

> layer, just as messages with unrecognized critical options are rejected.

I'm not sure I understand this point:

   o  Unrecognized options of class "critical" that occur in a
      Confirmable request MUST cause the return of a 4.02 (Bad Option)
      response.  

(5.4.1.  Yes, in other cases the message is rejected.  
But I was talking about the most prominent case here.)

>   reject any message with a larger option number.

Well, that latter "MUST" heaps another requirement on the receiver that costs money but that nobody will benefit from.

> the protocol.  If it doesn't, let's refine it.

I'd rather spend time refining observe, block (thank you for your recent suggestions on these), congestion control, CoRE AA, ...

Grüße, Carsten

---

From: "Peter A. Bigot" <[email-masked]>
Date: Mon, 21 Oct 2013 10:40:56 -0500

On 10/21/2013 09:32 AM, Carsten Bormann wrote:
> Data point: People have been able to build the web with a document as
> precise as RFC 2616. Only now, 15 years later, do we pick up the
> loose ends and try to write up httpbis (approximately doubling the
> number of pages).
>
> I'm not saying having a perfect specification would be undesirable,
> it's just that perfection here is empirically uncorrelated with the
> quality of the interoperability.

HTTP pre-dates RFC2616 by at least eight years that included huge
amounts of field deployment and interoperability.  CoAP is attempting to
define a specification based on (I'm guessing) a dozen or so coordinated
implementations being used in test environments.  RFC2616 leveraged
vastly more implementation experience and produced a document that is
far more precise than CoAP.

I'm not saying CoAP needs to be perfect from the start.  I'm saying we
need to help its adoption by accepting and addressing its flaws clearly
and visibly.  Doing that will help justify (or not) a future re-spin,
once there's actual field experience with interoperable systems that use it.

>> If this can't be resolved in coap-xx before it becomes an RFC, it
>> needs to be an erratum which can be found by direct link from the
>> IETF RFC pages.
>

No point arguing about hypotheticals.  If I'm still dabbling in CoAP
when the RFC comes out I'll report it as a technical erratum then so
future implementers are at least made aware of the potential issue.

> For documenting things like this, I think that RFC 4815 style
> documents work better than lists of errata.  Let's discuss in
> Vancouver how we want to handle this phase.

FWIW I won't be at Vancouver; my CoAP work is unfunded so travel isn't
an option.

>> Second, using 4.02 unnecessarily crosses a layer boundary.  That a
>> "big option" (one with a number above 65535) is even expressible is
>> due to a weakness in message encoding, and it should be diagnosed
>> at the message (CON/NON/ACK/RST) layer not at the REST transaction
>> (request/response) layer, just as messages with unrecognized
>> critical options are rejected.
>
> (5.4.1.  Yes, in other cases the message is rejected. But I was
> talking about the most prominent case here.)

OK, I can see that.  My search located the second clause:

    o  Unrecognized options of class "critical" that occur in a Non-
       confirmable message MUST cause the message to be rejected
       (Section 4.3).

and with wishful thinking I assumed that message-level errors would be
treated consistently.  The message-layer/transaction-layer distinction
is even weaker than I'd thought.

I have a very difficult time coming up with a mental model of how CoAP
is supposed to work, given how much of it is defined in terms of
behavior in specific cases instead of core rules.  The resulting
complexity of interactions between observe/block/groupcomm when mixing
message-layer and transaction-layer choices and expectations is huge.

Peter