Subject: [core] #331: No way to indicate preferred size in a 4.13 error

From: "core issue tracker" <[email-masked]>
Date: Wed, 12 Jun 2013 07:46:07 -0000

#331: No way to indicate preferred size in a 4.13 error

 Martin Stiemerling notes (msg04557):

 Section 4.6

 8) Flow Control/Receiver Buffer The protocol does not have any real means
 for the receiver to control the amount of data that is being sent to it. I
 can understand the attempt to provide a simple protocol, but adding a very
 basic flow control mechanism will not prohibitively increase the
 complexity of the protocol, while improving robustness. The 4.13 "Request
 Entity Too Large" with a protocol field that indicates the maximum data
 size would be fixing this.

 Carsten says:

 The design rules for CoAP would put such a "protocol field" into an Option
 (type uint) that would be sent back with the 4.13 response. This would be
 an elective option, as the 4.13 response is still valid information even
 if the option is not understood.

 Now, we already have two related mechanisms in -block:

 »The error code 4.13 (Request Entity Too Large) can be returned at any
 time by a server that does not currently have the resources to store
 blocks for a block-wise request payload transfer that it would intend to
 implement in an atomic fashion.  (Note that a 4.13 response to a request
 that does not employ Block1 is a hint for the client to try sending
 Block1, and a 4.13 response with a smaller SZX in its Block1 option than
 requested is a hint to try a smaller SZX.)«

 So we can use SZX in Block1 for the "datagram too large" situation.  Since
 that is only sent for a request that specified Block1, there may be two
 exchanges before a good datagram size is arrived at, if the requesting
 endpoint does support Block1, which is suboptimal, but probably acceptable
 for a multi-exchange Block1 transfer.

 With the Block1 case taken care of, we still don't have anything to
 indicate the desired size of the *entire* request payload. -block defines
 a Size option that is used for exchanging information about payload sizes.
 The Size option is currently artfully split between request and response
 semantics based on the presence of other options and some heuristics (just
 as Block was before it was split into Block1 and Block2).  Can we shoehorn
 in the required semantics, e.g. by giving it special semantics for the
 4.13 case?  Or should there be a Size1 and a Size2?

 And, independent of how exactly we solve this problem with -block: Is it a
 requirement that this meaning/variant of Size be included in the base
 protocol?  Or can we wait with this for a couple of months until -block is
 done?

-- 
-------------------------+-------------------------------------------------
 Reporter:               |      Owner:  [email-masked]
  [email-masked]           |     Status:  new
     Type:  protocol     |  Milestone:  post-IESG-1
  enhancement            |    Version:  coap-17
 Priority:  major        |   Keywords:
Component:  coap         |
 Severity:  Submitted    |
  WG Document            |
-------------------------+-------------------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/core/trac/ticket/331>
core <http://tools.ietf.org/core/>

---

From: "core issue tracker" <[email-masked]>
Date: Wed, 12 Jun 2013 08:07:50 -0000

Comment (by [email-masked]):

 From [1404]:

 Prepare ticket 331.

-- 
-------------------------+-------------------------------------------------
 Reporter:               |       Owner:  draft-ietf-core-
  [email-masked]           |  [email-masked]
     Type:  protocol     |      Status:  new
  enhancement            |   Milestone:  post-IESG-1
 Priority:  major        |     Version:  coap-17
Component:  coap         |  Resolution:
 Severity:  Submitted    |
  WG Document            |
 Keywords:               |
-------------------------+-------------------------------------------------

Ticket URL: </ticket/331#comment:1>
core <http://tools.ietf.org/core/>

---

From: "Rahman, Akbar" <[email-masked]>
Date: Sun, 16 Jun 2013 20:20:38 -0400

Hi Carsten,

With respect to your question:

"And, independent of how exactly we solve this problem with -block: Is it a  requirement that this meaning/variant of Size be included in the base  protocol?  Or can we wait with this for a couple of months until -block is  done?"

My vote is that, yes, we do solve it in the base draft (and not wait until -block is done).  I think Martin also wants the (or at least "a") solution in the base draft.

Best Regards,

Akbar

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of core issue tracker
Sent: Wednesday, June 12, 2013 3:46 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: [core] #331: No way to indicate preferred size in a 4.13 error

 Section 4.6

 8) Flow Control/Receiver Buffer The protocol does not have any real means  for the receiver to control the amount of data that is being sent to it. I  can understand the attempt to provide a simple protocol, but adding a very  basic flow control mechanism will not prohibitively increase the  complexity of the protocol, while improving robustness. The 4.13 "Request  Entity Too Large" with a protocol field that indicates the maximum data  size would be fixing this.

 Carsten says:

 The design rules for CoAP would put such a "protocol field" into an Option  (type uint) that would be sent back with the 4.13 response. This would be  an elective option, as the 4.13 response is still valid information even  if the option is not understood.

 »The error code 4.13 (Request Entity Too Large) can be returned at any  time by a server that does not currently have the resources to store  blocks for a block-wise request payload transfer that it would intend to  implement in an atomic fashion.  (Note that a 4.13 response to a request  that does not employ Block1 is a hint for the client to try sending  Block1, and a 4.13 response with a smaller SZX in its Block1 option than  requested is a hint to try a smaller SZX.)«

 So we can use SZX in Block1 for the "datagram too large" situation.  Since  that is only sent for a request that specified Block1, there may be two  exchanges before a good datagram size is arrived at, if the requesting  endpoint does support Block1, which is suboptimal, but probably acceptable  for a multi-exchange Block1 transfer.

 With the Block1 case taken care of, we still don't have anything to  indicate the desired size of the *entire* request payload. -block defines  a Size option that is used for exchanging information about payload sizes.
 The Size option is currently artfully split between request and response  semantics based on the presence of other options and some heuristics (just  as Block was before it was split into Block1 and Block2).  Can we shoehorn  in the required semantics, e.g. by giving it special semantics for the
 4.13 case?  Or should there be a Size1 and a Size2?

 And, independent of how exactly we solve this problem with -block: Is it a  requirement that this meaning/variant of Size be included in the base  protocol?  Or can we wait with this for a couple of months until -block is  done?

-- 
-------------------------+----------------------------------------------
-------------------------+---
 Reporter:               |      Owner:  [email-masked]
  [email-masked]           |     Status:  new
     Type:  protocol     |  Milestone:  post-IESG-1
  enhancement            |    Version:  coap-17
 Priority:  major        |   Keywords:
Component:  coap         |
 Severity:  Submitted    |
  WG Document            |
-------------------------+----------------------------------------------
-------------------------+---

_______________________________________________
core mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: "Dijk, Esko" <[email-masked]>
Date: Mon, 17 Jun 2013 06:46:47 +0000

> My vote is that, yes, we do solve it in the base draft (and not wait until -block is done).  I think Martin also wants the (or at least "a") solution in the base draft.
Agree, it looks like Martin's request is to have it solved in the base draft. (Whether now or in a few months time.)

But of course the choice of a 'Size' option or 'Size1/Size2' options solution may be linked to core-block which now defines the Size option.
An elective Size/Size* option would be best to solve this with minimal complexity impact on the client.

(And: MUST or SHOULD or MAY a CoAP server send the Size option back  with a 4.13...?)

Esko

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rahman, Akbar
Sent: Monday, June 17, 2013 02:21
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Hi Carsten,

Best Regards,

Akbar

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of core issue tracker
Sent: Wednesday, June 12, 2013 3:46 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]
Subject: [core] #331: No way to indicate preferred size in a 4.13 error

 Section 4.6

 Carsten says:

________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Jun 2013 18:34:56 +0200

> Agree, it looks like Martin's request is to have it solved in the base draft. (Whether now or in a few months time.)

Yes, and our job as a WG is to find out whether that is a reasonable request and, if yes, implement it (or, if no, supply good arguments for why not).

> An elective Size/Size* option would be best to solve this with minimal complexity impact on the client.

Right.  So here is my proposal:

-- We update -block to split Size into Size1 and Size2.

The current text says:

    »The Size Option may be used for three purposes:

    * in a request, to ask the server to provide a size estimate along
      with the usual response  ("size request").  For this usage, the value MUST be set to
      0.
    * in a response carrying a Block2 Option, to indicate the current
      estimate the server has of the total size of the resource
      representation.
    * in a request carrying a Block1 Option, to indicate the current
      estimate the client has of the total size of the resource
      representation.

    In the latter two cases ("size indication"), the value of the option
    is the current estimate, measured in bytes.

    A size request can be easily distinguished from a size indication, as
    the third case is not useful for a GET or DELETE, and an
    actual size indication of 0 would either be overridden by the actual
    size of the payload for a PUT or POST or would not be useful.«

Adding the 4.13 feedback semantics as another special case to this
slighly cluttered usage of a single is theoretically possible, but
becomes harder and harder to justify.

So, instead, we start to distinguish a Size1, which is always about
the resource representation in the request, and Size2, which is always
about the one in the response.  Both are elective, safe-to-forward,
no-cache-key.  We can get rid of the "easily distinguished" text above.

For handling the discuss, we only need Size1.
I'm assuming most people who have implemented Size use it in Size2
semantics, so a minimum change addition to -block is:

| Type | C | U | N | R | Name  | Format | Length | Default |
|------+---+---+---+---+-------+--------+--------+---------|
|   60 | - | - | N | - | Size1 | uint   | 0-4 B  | (none)  |
|   28 | - | - | N | - | Size2 | uint   | 0-4 B  | (none)  |

(The somewhat weirdly high option number 60 = 0b1_111_00 results from
us only allocating 1/8 of the safe-to-forward space to no-cache-key.)

-- We introduce the Size1 option into the base spec.

Add to table 4 in 5.10:

| Type | C | U | N | R | Name  | Format | Length | Default |
|------+---+---+---+---+-------+--------+--------+---------|
|   60 |   |   | x |   | Size1 | uint   |    0-4 | (none)  |

(Yes, we have to adopt the updated table notation from -coap in -block
as well at some point.)

Add a new section 5.10.9:

The Size1 option provides size information about the resource
representation in a request.  The option value is an integer number of
bytes.  Its main use is with block-wise transfers
[I-D.ietf-core-block].  In the present specification, it is used in
4.13 responses to indicate the maximum size of request entity that the
server is able and willing to handle.

Add to the end of 5.9.2.9.  4.13 Request Entity Too Large:

The response SHOULD include a Size1 Option to indicate the maximum
size of request entity the server is able and willing to handle,
unless the server is not in a position to make this information
available.

                                 oOo

Is this the right thing to do?
Is the SHOULD the right level of requirement?
Do people think they would be implementing this?
Are there any obstacles to implementation we haven't thought about?

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 17 Jun 2013 20:26:19 +0300

Regarding if this should be handled in the base CoAP specification or Block…. My strong preference would be to do this in Block, as it is not useful (and could be confusing) in the base CoAP specification.

I don't see any practical reasons or use cases for having a Size field without the use of Block transfer. CoAP is a lock-step protocol with a typical maximum message size of 1152 bytes. IPv6 devices are required to support the minimum MTU, therefore there is really no use for size feedback. 

Now, I do see several applications with Block transfer, both when a node prefers very small blocks to avoid 6LoWPAN fragmentation or when limited buffer space is available for larger blocks. Considering the interactions between Size1, Size2 and 4.13, that level of specification is anyways best done in Block.

Zach  

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com @SensinodeIoT
Mobile: +358 40 7796297
Twitter: @zach_shelby
LinkedIn: http://fi.linkedin.com/in/zachshelby
6LoWPAN Book: http://6lowpan.net

---

From: "Rahman, Akbar" <[email-masked]>
Date: Mon, 17 Jun 2013 15:30:30 -0400

Hi Zach,

Just to clarify and sync with you.  I understood that Martin's 4.13 related comment was made on section 4.6 (Message Size) which is where the 4.13 processing is discussed (in an Implementation Note).  In that section it says:   

      "Implementations receiving a datagram into a buffer that is too
      small are usually able to determine if the trailing portion of a
      datagram was discarded and to retrieve the initial portion.  So,
      if not all of the payload, at least the CoAP header and options
      are likely to fit within the buffer.  A server can thus fully
      interpret a request and return a 4.13 (Request Entity Too Large,
      see Section 5.9.2.9) response code if the payload was truncated.
      A client sending an idempotent request and receiving a response
      larger than would fit in the buffer can repeat the request with a
      suitable value for the Block Option [I-D.ietf-core-block]."

Then Martin goes on to suggest that:

     "The 4.13 "Request  Entity Too Large" with a protocol field that indicates the maximum data size would be fixing this."

So, actually, according to the current text it does seem like you could have a problem before Block was invoked.  Do you agree?

Best Regards,

Akbar

-----Original Message-----
From: Zach Shelby [mailto:[email-masked]] 
Sent: Monday, June 17, 2013 1:26 PM
To: Carsten Bormann
Cc: Dijk, Esko; Rahman, Akbar; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Regarding if this should be handled in the base CoAP specification or Block.... My strong preference would be to do this in Block, as it is not useful (and could be confusing) in the base CoAP specification.

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Jun 2013 21:32:20 +0200

On Jun 17, 2013, at 21:30, "Rahman, Akbar" <[email-masked]> wrote:

> you could have a problem before Block was invoked

Yes, but how would you solve it except by using Block or a block-like protocol?

Grüße, Carsten

---

From: "Rahman, Akbar" <[email-masked]>
Date: Mon, 17 Jun 2013 15:43:37 -0400

Well, at least the Client could stop trying to re-send the payload if it knew the maximum allowable size of the payload but was not willing/able to reduce it.  This would at least throttles (stops) the transaction that was causing congestion.  That is of course the simplest (brute force) flow control mechanism you can design.  

But it at least solves one end of the spectrum of flow control that Martin was worried about.  The more sophisticated solution of course will be in Block but the point is that is that the base CoAP spec should be self-consistent and not require Block to at least stop simple congestion scenarios.

Best Regards,

Akbar

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: Monday, June 17, 2013 3:32 PM
To: Rahman, Akbar
Cc: Zach Shelby; Dijk, Esko; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Jun 2013 22:19:15 +0200

On Jun 17, 2013, at 21:43, "Rahman, Akbar" <[email-masked]> wrote:

> Well, at least the Client could stop trying to re-send the payload if it knew the maximum allowable size of the payload but was not willing/able to reduce it.  

That is already covered by 4.13 as it is -- after all the client has received an error and has no reason to believe a retry will fix it (that is the general semantics of a 4.xx).

Adding a Size option would allow the client to re-try while reducing the request payload size based on information from the server.

But, I'm still curious: what would be the use case for such a reduction outside of a block-wise protocol?

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 17 Jun 2013 23:29:52 +0300

> But, I'm still curious: what would be the use case for such a reduction outside of a block-wise protocol?

That is exactly my point, I haven't been able to find a use case for this (size) without block. 

Zach

---

From: "Rahman, Akbar" <[email-masked]>
Date: Mon, 17 Jun 2013 16:37:52 -0400

There is nothing in the current text (I think) to stop a silly client from retrying to send the payload (with old-size -1) after receiving a 4.13 error (and then keep looping if that also results in another 4.13 error).  I think we have all seen silly implementations like this in other protocol implementations.  

I am not trying to justify this behavior, I am just saying that this is what I think that Martin means by a simple fix.

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: Monday, June 17, 2013 4:19 PM
To: Rahman, Akbar
Cc: Zach Shelby; Dijk, Esko; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Grüße, Carsten

---

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 18 Jun 2013 05:55:47 +0000

Use cases can always be found  ;)

One is a client using the "Batch" concept from core-interfaces (http://tools.ietf.org/html/draft-ietf-core-interfaces-00#section-5.2 ) to update (PUT) multiple resources. A 4.13 is returned, so the client decides to update the individual resources with multiple PUTs instead.

Esko

-----Original Message-----
From: Zach Shelby [mailto:[email-masked]]
Sent: Monday, June 17, 2013 22:30
To: Carsten Bormann
Cc: Rahman, Akbar; Dijk, Esko; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Zach

---

From: Zach Shelby <[email-masked]>
Date: Tue, 18 Jun 2013 09:10:24 +0300

> One is a client using the "Batch" concept from core-interfaces (http://tools.ietf.org/html/draft-ietf-core-interfaces-00#section-5.2 ) to update (PUT) multiple resources. A 4.13 is returned, so the…

But why would the endpoint even support the Batch resource type and a batch media type such as SenML if it can't process it? Instead the endpoint would only make individual text/plain resources available in the first place, so the client knows not to try and use Batch. 

Zach

---

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 18 Jun 2013 06:29:13 +0000

My approach is to let's think of the best use cases we can find for this situation and then see whether they make sense. (If not, no solution to the problem is needed probably?)

The server in this use case would support batch of course, but it has a buffer size that is smaller (e.g. 768 bytes) than the size the client expects (e.g. 1024 bytes payload size based on IPv6 MTU).
The batching of resources works as long as the CoAP request payload is 768 bytes or less. Normally it's used to batch only small resources. But, now the client, not knowing the 768 byte limit, simply puts a few more large entries in the batch request, so the request payload size grows above this limit to e.g. 1000 bytes.
Then, the server would presumably (http://tools.ietf.org/html/draft-ietf-core-coap-17#section-4.6) return a 4.13 response.

To me this sounds like a situation that could very well happen. And there's no mandate in CoAP to use core-block if payload size exceed ~ 100 bytes.

Esko

-----Original Message-----
From: Zach Shelby [mailto:[email-masked]]
Sent: Tuesday, June 18, 2013 08:10
To: Dijk, Esko
Cc: Carsten Bormann; Rahman, Akbar; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Jun 2013 15:21:14 +0200

My personal summary of the conversation on this ticket so far:

-- We don't have consensus that the solution solves a particularly prominent use case.
-- We don't have consensus that it is entirely useless either.
-- Some of the discussion was about whether we do it now or in -block, which is just a matter of procedure.
-- Nobody has commented on the specific solution I have proposed.

Given that this is one of the two remaining issues blocking CoAP from becoming RFC, I'm inclined to just make the change.  But I'd like to hear a bit more about whether the other elements of the proposed change (the ones that also influence -block) are right:

-- Splitting Size into Size1 and Size2 (and removing some of the heuristic stuff that we need without that);
-- Using Size1 not only in a request to indicate the projected size of the request body, but also in a 4.13 response to indicate the maximum accepted size.

I would like to submit -18 very soon, so I would be happy with some quick responses, even if they are only temperature readings.

Grüße, Carsten

---

From: "Dijk, Esko" <[email-masked]>
Date: Sun, 23 Jun 2013 12:09:01 +0000

Hi Carsten,

the proposed changes sound ok. I can't judge whether the 'heuristic stuff' you mention leads to code complexity today, or not. It looks as if using Size1/Size2 instead of a single 'Size' option would not matter that much for current implementations. Maybe it is important though to be prepared better for future extensions of CoAP (just like split between Block1/Block2 became necessary at some point to support the POST case.)

Esko

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]]
Sent: Friday, June 21, 2013 15:21
To: Dijk, Esko
Cc: Zach Shelby; Rahman, Akbar; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Grüße, Carsten

---

From: "Rahman, Akbar" <[email-masked]>
Date: Sun, 23 Jun 2013 18:36:32 -0400

+1

-----Original Message-----
From: Dijk, Esko [mailto:[email-masked]] 
Sent: Sunday, June 23, 2013 8:09 AM
To: Carsten Bormann
Cc: Zach Shelby; Rahman, Akbar; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [core] #331: No way to indicate preferred size in a 4.13 error

Hi Carsten,

Esko

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]]
Sent: Friday, June 21, 2013 15:21
To: Dijk, Esko
Cc: Zach Shelby; Rahman, Akbar; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [core] #331: No way to indicate preferred size in a 4.13 error

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 24 Jun 2013 10:05:52 +0300

Carsten,

I am fine with just implementing this change.

Zach

---

From: "core issue tracker" <[email-masked]>
Date: Thu, 27 Jun 2013 13:18:17 -0000

Changes (by [email-masked]):

 * status:  new => closed
 * resolution:   => fixed

Comment:

 Fixed in [1413]:

 Fix #331

-- 
-------------------------+-------------------------------------------------
 Reporter:               |       Owner:  draft-ietf-core-
  [email-masked]           |  [email-masked]
     Type:  protocol     |      Status:  closed
  enhancement            |   Milestone:  post-IESG-1
 Priority:  major        |     Version:  coap-17
Component:  coap         |  Resolution:  fixed
 Severity:  Submitted    |
  WG Document            |
 Keywords:               |
-------------------------+-------------------------------------------------

Ticket URL: </ticket/331#comment:13>
core <http://tools.ietf.org/core/>