Subject: Re: [core] Apparent race condition in draft-ietf-core-block-08

From: Robert Quattlebaum <[email-masked]>
Date: Sat, 2 Feb 2013 11:46:03 -0800

Hello Maciej. Sorry for the late reply.

On Jan 23, 2013, at 12:31 PM, Maciej Wasilak <[email-masked]> wrote:

> As far as I understand all stateful blockwise transfers should be identified now by a tuple (client, uri_path), where client is another tuple (address, port). I agree that introducing proxies might lead to race conditions. It is true for PUT/POST transfers, but also for stateful GET transfers (as in draft-ietf-core-block-10: "The server may identify the sequence by the combination of the requesting end-point and the URI being the same in each block-wise request."). So if two GET requests (for example with different uri-query) are sent through a proxy, server also won't be able to tell them apart. 

This is true as well, but with GET transfers there are mechanisms
that can be used to work around this, such as caching. However, a
non-caching proxy server could easily run afoul of this problem with
a blockwise GET.

> Some practical workaround might be sending POST/PUT requests to subresources instead of a base resource: 
> POST /data-sink/sensor1 
> POST /data-sink/sensor2
> where sensor1 and sensor2 are temporary labels, instead of:
> POST /data-sink
> However both proxy and server need to understand these labels, which might be complicated.

Yes, you could work around this issue at the application level, but
it would be profoundly unfortunate for such a hack to be necessary
in the first place. If we have the opportunity to truly fix this
at this level before the drafts become standards, I think we should.
An ideal solution would require no changes for transfers which
disambiguation is not necessary.

I have bounced a few ideas for solutions off of Carsten, but he
seems to be rather busy lately. The solution I brought up was to
add a new option called a 'stoken'. The solution has the following
pros and cons:

### Pros ###

 * Appears to solve all of the problems I described earlier.
 * Has additional benefits for constrained servers serving dynamic
   content.
 * Client doesn't need prior knowledge if the option is necessary
   because the client only uses the option when told to by the server.
 * Works with both block1 and block2.
 * Not conceptually limited to the block protocol. For example, it
   could be used for canceling individual observations (although that
   would only really be a problem if you were using a non-caching
   proxy as far as I can tell)
 * Option would only be used by the server when required, so
   constrained clients may leave it out and still continue to work
   in the majority of cases.
 * Constrained Clients which don't implement the stoken option (and
   have an NSTART of 1) could still benefit from the use of the option
   by using a proxy which supported the stoken option.

### Cons ###

 * Adds yet another option.
 * Clients wanting to implement this would need to be able to store
   an additional ~10 bytes of state between receipt of a response and
   the sending of the next related request.

A more formalized explanation of the semantics of the option are described
below. I'd love some feedback, especially about any potential edge cases or
other issues.

--------------------

 * Option Name: `Stoken`
 * Option Number: TBD
 * Critical: TBD
 * Unsafe: No
 * Data: Opaque 0-8 bytes
 * Default value: Empty

The `stoken` option is used when a server needs to associate a
subsequent request as being definitively related to a specific prior
request. The specific value of the stoken option is meaningful only
to the server and is entirely opaque to the client. Only sequential
requests can be related using this mechanism.

If a server receives a request that it needs to definitively associate
with a subsequent request, it will add a stoken option to the
response (the specific value of which is implementation specific).
When the client sends the next request in the series, it will include
the stoken option from the previously received response. The server can
then use the stoken value to relate the current request to the
previous request. This process can associate an indefinite number
of requests. The server is not restricted to using the same stoken
value for each response in the series.

  CLIENT                                         SERVER
    |                                              |
    | CON [MID=1234], GET, /status         ------> |
    |                                              |
    | <------   ACK [MID=1234], 2.05 Content       |
    |           Stoken:5678 Block2:0/1/128         |
    |                                              |
    | CON [MID=1235], GET, /status         ------> |
    | Stoken:5678 Block2:1/1/128                   |
    |                                              |
    | <------   ACK [MID=1235], 2.05 Content       |
    |           Stoken:9012 Block2:1/1/128         |
    |                                              |
    | CON [MID=1236], GET, /status         ------> |
    | Stoken:9012 Block2:2/0/128                   |
    |                                              |
    | <------   ACK [MID=1236], 2.05 Content       |
    |           Block2:2/0/128                     |
            Figure 1. Block2 with Stoken

  CLIENT                                         SERVER
    |                                              |
    | CON [MID=1237], PUT, /options        ------> |
    | Block1:0/1/128                               |
    |                                              |
    | <------   ACK [MID=1237], 2.05 Content       |
    |           Stoken:1111 Block1:0/1/128         |
    |                                              |
    | CON [MID=1238], PUT, /options        ------> |
    | Stoken:1111 Block1:1/1/128                   |
    |                                              |
    | <------   ACK [MID=1238], 2.05 Content       |
    |           Stoken:2222 Block1:1/1/128         |
    |                                              |
    | CON [MID=1239], PUT, /options        ------> |
    | Stoken:2222 Block1:2/0/128                   |
    |                                              |
    | <------   ACK [MID=1239], 2.05 Content       |
    |           Block1:1/0/128                     |
            Figure 2. Block1 with Stoken

From a different perspective, the stoken option may be used to store
server state about the ongoing transaction, without actually storing
any state in the server. This is especially useful when serving
potentially large content---like link-formats---dynamically, without
server-side state.

Example: Let's say we have a large list of links that is dynamically
generated. Each link entry is a variable size. With the stoken
parameter, we can implement this efficiently without any state on
the server by storing the index of the last item along with the
number of bytes from the last item we managed to add to the end of
the returned block. When the client sends the request for the next
block, we will know exactly where we left off from. Without the
stoken parameter we would have either had to start over from scratch
on each request (dramatically increasing the resources required to
serve later requests) or store the state for each transaction, keyed
off of the client IP address and port (Which I have just shown to be
somewhat unreliable).

-- Robert

> 
> Best Regards
> 
> Maciej Wasilak
> 2012/12/24 Robert Quattlebaum <[email-masked]>
>> Hello everyone,
>> 
>> I just wanted to bring up a race condition apparently present in
>> the current blockwise transfer draft. However, even if you are not
>> interested in draft-ietf-core-block, you may want to read over this
>> email anyway: Similar problems may be present in protocols which
>> need the server to associate multiple messages as being a part of
>> the same logical request. For now I'll limit the discussion to
>> draft-ietf-core-block-10.
>> 
>> Previously, the token field (as well as the source IP address and
>> port) could be used by the server to correlate multiple messages
>> that were a part of the same logical request ("token-affinity").
>> In this sense the token field was not really opaque because the
>> server was relying on the value of the token to not change between
>> subsequent messages in the same logical request. However, this
>> semantic-entanglement had the benefit of ensuring truly atomic
>> behavior of requests which consist of multiple messages, without
>> depending on artificial limitations on how the protocol should be
>> used or the behavior of a specific transport layer.
>> 
>> At some point over the past year (Forgive me, I haven't been paying
>> attention), the semantics of the token parameter was clarified to
>> be entirely opaque (meaning that only the client's address/port
>> could be used for determining which messages are a part of the same
>> blocked request). This was a reasonable change because it allows
>> client implementations to be more flexible. However, without further
>> clarification or additional mechanisms, this change leads to
>> undefined/unreliable behavior in certain cases.
>> 
>> Consider the case where you have a LoWPAN where nodes regularly
>> dump gathered information to a CoAP server in "the cloud" via a
>> CoAP-CoAP proxy. If two (or more) nodes try to perform a large POST
>> (using Block1) to the same CoAP URL thru the proxy at the same time,
>> the destination machine in the cloud has no way to differentiate
>> the messages as being two independent atomic requests.
>> 
>> In a previous discussion off-list, Carsten suggested that the proxy
>> could identify the start of subsequent requests as potentially
>> conflicting and use a different source port for each conflicting
>> request sent from the proxy, but this technique only works if the
>> cloud-side transport layer uses the UDP/IP transport---this technique
>> would not work if the proxy was sending requests to the cloud via
>> CoAP-over-SMS, as described in section 14 of
>> draft-becker-core-coap-sms-gprs-02.
>> 
>> I think the root of this problem is the reliance on information
>> from the transport layer in determining which individual messages
>> belong to the same logical request. Doing this just happens to work
>> out in most cases. However, because these edge cases are transient
>> in nature, diagnosing such problems in the field may be very difficult
>> and frustrating. Sure, there are work-arounds, but they have specific
>> disadvantages which may not be obvious.
>> 
>> The bottom line is that by refining the semantics of the token field
>> to no longer allow the server to use token-affinity to associate
>> multiple messages to logical requests, a race condition has been
>> introduced into the protocol. This race condition was hidden by the
>> fact that, by using the address/port information from the transport
>> layer, things *usually* work just fine. Usually, but not always.
>> 
>> All that being said, I don't necessarily believe that reverting the
>> token behavior is necessarily the right solution to this problem:
>> Allowing the client to use the token to keep track of state on a
>> message-per-message basis is useful. The trick is giving the same
>> flexibility to the server: If we can't rely entirely on the transport
>> layer information to associate logically-related messages (and we
>> can't use the token) then what should be used?
>> 
>> We could always simply document this class of problems in the draft
>> so that implementors are aware that they may need to work around
>> it, but I worry that this may be the first of a class of similar
>> race conditions if this issue is not addressed directly.
>> 
>> Thoughts?
>>

---

From: "Kovatsch Matthias" <[email-masked]>
Date: Wed, 6 Feb 2013 16:21:03 +0000

Dear all

I see this race condition as a problem that should be fixed in a cleaner way than providing workarounds like an additional option or even temporary resources. This stoken would enable cookies for CoAP, which might go a bit too far for constrained environments. The mentioned keeping of state is already provided in a limited way through the byte offset provided by the block options. But maybe there are other scenarios for cookies?

The main reason to disentangle the token and block options was that it did appear "obscure" that a client would perform multiple concurrent PUT/POST requests. The token dependency only introduced complexity that we could get rid of. In Robert's scenario, however, concurrent transfers is exactly what the proxy would do in its client role (let's assume congestion control is obeyed); and it does make sense for POSTs that result in the creation of new resources. The natural thing would be that the proxy assigns different tokens to the open requests, so the proxy can also distinguish the responses and send them back to the correct clients.

This would mean that we need to revert back to the (IP, port, Uri-Path, token) tuple to identify ongoing transfers. An alternative would be to put more complexity onto the proxy: it would have to cache and serialize the incoming requests.

I believe the origin server must have some means to ensure its resources do not corrupt in normal operation and to decide if they did, e.g., because of a client crash in the middle of a transfer. For more powerful servers, this is atomic processing. For constrained servers, I would (in Erbium I actually do) only allow one ongoing transfer and send a 5.03 "Service Unavailable" response to all others. Both will need a timeout to detect incomplete transfers. This goes in hand with reverting to the use of the token, as origin servers cannot trust proxies to get the serialization right. Are there any objections, comments, or agreements on that?

Not corrupting resources is also related to the random access feature through blockwise transfers. For Block2 it enables a resume feature for free, so that is okay. But for Block1, this feature does not make much sense in my eyes. One would always have to deal with padding like whitespaces in JSON to have a random access behavior. It makes much more sense to use queries to access specific parts of a resource!

Thus, I still request to enforce in-order transfers of blocks, always starting at zero for Block1.

Ciao
Matthias

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Quattlebaum
> Sent: Samstag, 2. Februar 2013 20:46
> To: Maciej Wasilak
> Cc: IETF CoRE Working Group
> Subject: Re: [core] Apparent race condition in draft-ietf-core-block-08
> 
> > As far as I understand all stateful blockwise transfers should be identified
> now by a tuple (client, uri_path), where client is another tuple (address,
> port). I agree that introducing proxies might lead to race conditions. It is true
> for PUT/POST transfers, but also for stateful GET transfers (as in draft-ietf-
> core-block-10: "The server may identify the sequence by the combination of
> the requesting end-point and the URI being the same in each block-wise
> request."). So if two GET requests (for example with different uri-query) are
> sent through a proxy, server also won't be able to tell them apart.
> 
> This is true as well, but with GET transfers there are mechanisms that can be
> used to work around this, such as caching. However, a non-caching proxy
> server could easily run afoul of this problem with a blockwise GET.
> 
> Yes, you could work around this issue at the application level, but it would be
> profoundly unfortunate for such a hack to be necessary in the first place. If
> we have the opportunity to truly fix this at this level before the drafts
> become standards, I think we should.
> I have bounced a few ideas for solutions off of Carsten, but he seems to be
> rather busy lately. The solution I brought up was to add a new option called a
> 'stoken'. The solution has the following pros and cons:
> 
> ### Pros ###
> 
> The `stoken` option is used when a server needs to associate a subsequent
> request as being definitively related to a specific prior request. The specific
> value of the stoken option is meaningful only to the server and is entirely
> opaque to the client. Only sequential requests can be related using this
> mechanism.
> 
> If a server receives a request that it needs to definitively associate with a
> subsequent request, it will add a stoken option to the response (the specific
> value of which is implementation specific).
> When the client sends the next request in the series, it will include the
> stoken option from the previously received response. The server can then
> use the stoken value to relate the current request to the previous request.
> This process can associate an indefinite number of requests. The server is not
> restricted to using the same stoken value for each response in the series.
> 
>   CLIENT                                         SERVER
>     |                                              |
> From a different perspective, the stoken option may be used to store server
> state about the ongoing transaction, without actually storing any state in the
> server. This is especially useful when serving potentially large content---like
> link-formats---dynamically, without server-side state.
> 
> Example: Let's say we have a large list of links that is dynamically generated.
> Each link entry is a variable size. With the stoken parameter, we can
> implement this efficiently without any state on the server by storing the
> index of the last item along with the number of bytes from the last item we
> managed to add to the end of the returned block. When the client sends the
> request for the next block, we will know exactly where we left off from.
> Without the stoken parameter we would have either had to start over from
> scratch on each request (dramatically increasing the resources required to
> serve later requests) or store the state for each transaction, keyed off of the
> client IP address and port (Which I have just shown to be somewhat
> unreliable).
> 
> -- Robert
> 
> >
> > Best Regards
> >
> > Maciej Wasilak
> >> I just wanted to bring up a race condition apparently present in the
> >> current blockwise transfer draft. However, even if you are not
> >> email anyway: Similar problems may be present in protocols which need
> >> the server to associate multiple messages as being a part of the same
> >> logical request. For now I'll limit the discussion to
> >> port) could be used by the server to correlate multiple messages that
> >> were a part of the same logical request ("token-affinity").
> >> attention), the semantics of the token parameter was clarified to be
> >> entirely opaque (meaning that only the client's address/port could be
> >> used for determining which messages are a part of the same blocked
> >> request). This was a reasonable change because it allows client
> >> implementations to be more flexible. However, without further
> dump
> >> gathered information to a CoAP server in "the cloud" via a CoAP-CoAP
> >> proxy. If two (or more) nodes try to perform a large POST (using
> >> Block1) to the same CoAP URL thru the proxy at the same time, the
> >> destination machine in the cloud has no way to differentiate the
> >> messages as being two independent atomic requests.
> >>
> >> I think the root of this problem is the reliance on information from
> >> the transport layer in determining which individual messages belong
> >> to the same logical request. Doing this just happens to work out in
> >> most cases. However, because these edge cases are transient in
> >> nature, diagnosing such problems in the field may be very difficult
> >> so that implementors are aware that they may need to work around it,
> >> but I worry that this may be the first of a class of similar race
> >> conditions if this issue is not addressed directly.
> >>
> >> Thoughts?
> >>
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Robert Quattlebaum <[email-masked]>
Date: Wed, 6 Feb 2013 23:04:02 -0800

Hello Kovatsch,

I just wanted to clarify one point:

> I see this race condition as a problem that should be fixed in a cleaner way than providing workarounds like an additional option or even temporary resources. This stoken would enable cookies for C…

Cookies, to me anyway, imply storage and potentially other complex mechanisms. The `stoken` option makes for a lousy cookie. It's only purpose is to relate different requests as being apart of a single logical operation. If present in a server response, it is used only once in a subsequent related request (if any). After that it is discarded (or replaced with a new stoken).

-- Robert