Subject: [core] Option method/response validation

From: "Peter A. Bigot" <[email-masked]>
Date: Fri, 04 Oct 2013 18:18:41 -0500

Section 5.4 of coap-18 has:

   Not all options are defined for use with all methods and response
   codes.  The possible options for methods and response codes are
   defined in Section 5.8 and Section 5.9 respectively.  In case an
   option is not defined for a method or response code, it MUST NOT be
   included by a sender and MUST be treated like an unrecognized option
   by a recipient.

The use of MUST (NOT) here is a strong (technically "absolute")
requirement, but I'm not finding much guidance from 5.8 and 5.9
regarding which options are defined for which methods and response
codes. This makes meeting the requirement difficult.

Any comments on the following tentative conclusions?

If-Match and If-None-Match are permitted for PUT by 5.8.3, but not
mentioned in other request methods. I don't think they make sense for
GET; I can imagine a use for POST; and If-Match (but not If-None-Match)
certainly makes sense for DELETE. Disallow for GET, allow all others
except If-None-Match for DELETE is disallowed.

Uri-Host, Uri-Port, Uri-Path, and Uri-Query may appear in any request,
and in no responses.

Location-Path and Location-Query may only appear in 2.01 Created
responses.

Content-Format is explicitly allowed only for 0.03 PUT. It is
implicitly allowed for 0.02 POST requests and 2.xx Success responses
except for 2.03 ValidContent because these codes mention a
representation in the payload and 5.5.1 says Content-Format applies to
resource and action result payloads. 5.5.2 implies it must not appear
for a diagnostic payload, therefore cannot appear in 4.xx or 5.xx error
responses.

Accept can appear only in 0.01 GET requests. But the other methods all
have valid responses that can contain non-diagnostic content, so maybe
it's ok for them too?

These options cannot appear (except as unrecognized) in any request:
Location-Path, Max-Age, Location-Query, Size1 (unless block is supported)

These options cannot appear (except as unrecognized) in any response:
If-Match, Uri-Host, If-None-Match, Uri-Port, Uri-Path, Uri-Query, 
Accept, Proxy-Uri, Proxy-Scheme

I'd appreciate feedback if these conclusions are thought to be wrong, as
well as insight on how much people expect extensions to be
allowed to change the restrictions for existing options (as Size1 would
be permitted in (some?) requests under draft-ietf-core-block-12). As an
example, I'd love to be able to put Uri-* in a future response related
to redirection, even though default Uri-Host and Uri-Post values would
have to be defined for response use in terms of the source endpoint.

Peter

---

From: Thomas Fossati <[email-masked]>
Date: Sat, 5 Oct 2013 08:53:16 +0100

Hi Peter,

On Sat, Oct 5, 2013 at 12:18 AM, Peter A. Bigot <[email-masked]> wrote:
> GET;

one possible use for GET+If-None-Match is to implement a sort of HEAD:
if the server has the requested resource, then it could respond 2.03
with the Etag'ed representation metadata?

---

From: "Peter A. Bigot" <[email-masked]>
Date: Sat, 05 Oct 2013 07:20:32 -0500

> with the Etag'ed representation metadata?

I think this would change conditional request too much.

If-None-Match succeeds only if the resource has no representation 
(interpreted to mean it does not exist on the server), so an 
unconditional GET would return 4.04 Not Found.  5.10.8 says this means 
the If-None-Match may be ignored in this case.

If-Match with an empty string would match any representation, so is a 
test for existence of the resource.  The current representation might 
not even have an entity-tag; and if it does there's no reasonable way 
for a client to ask what an arbitrary entity-tag represents, so the 
server could only return 2.03 Valid if the request included the ETag to 
tell the server it knew about that particular representation.  But then 
this is functionally equivalent to an unconditional GET.

However: DELETE with If-None-Match would return 4.12 Precondition Failed 
if the resource existed (as opposed to 2.05 Deleted or an authorization 
error if the conditional option was absent), and 4.04 Not Found if it 
didn't, so that actually already is a sort of HEAD low-cost probe for 
existence without returning the representation.  (Unless the server may 
check for authorization data first and return 4.03 Forbidden even if the 
resource doesn't exist, which case you can't tell anything from the result.)

Interesting.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 5 Oct 2013 16:20:38 +0200

> except If-None-Match for DELETE is disallowed.

I read the mention in 5.8.3 as a helpful pointer, not as Expressio unius est exclusio alterius.

5.10.8 makes no attempt at restricting these options to specific methods.
(5.10.8.2 mentions some combinations are "not very useful", but doesn't attempt to rule them out.)

> Uri-Host, Uri-Port, Uri-Path, and Uri-Query may appear in any request,

Indeed.
(They MUST NOT be included in conjunction with a Proxy-Uri Option, see 5.10.2.)

> Location-Path and Location-Query may only appear in 2.01 Created

At least that is the only usage defined at the moment.
(If we do go for a redirection extension, that would likely use these, too, and not the Uri-* options.)

> for a diagnostic payload, therefore cannot appear in 4.xx or 5.xx error

That is not exactly what it says.

5.5.1 says that whenever the payload is a representation, the Content-Format option gives you its media type (and that applies to all methods and response codes).  It then weasels around the ability to leave off the option when the application can divine its value.  The end of 5.5.1 combined with 5.5.2 says that if it is left off with a 4.xx/5.xx response, any payload present is not a representation but a diagnostic payload.  So there is no way to have a "format divined by application" representation with 4.xx/5.xx, but you can very well send a representation with these response codes, which is indicated by giving a Content-Format option.

The diagnostic payload (read "not a representation") was invented as a way to transport the "reason phrase" of an HTTP status line where it most matters, which is in the error cases.

> Accept can appear only in 0.01 GET requests.

I don't read 5.5.4 that way at all; it is actually quite useful to be able to select the content-format of a POST response.
(Again, there is a helpful pointer in 5.8.1.)

> But the other methods all

Yes.

> Location-Path, Max-Age, Location-Query, Size1 (unless block is supported)

(Max-Age had request semantics in an earlier version of CoAP, we removed that.)

I don't think including a Size1 option in a request is outlawed by 5.10.9, it is just not very useful without -block, as it is redundant with the size of the actual payload supplied.

> If-Match, Uri-Host, If-None-Match, Uri-Port, Uri-Path, Uri-Query, Accept, Proxy-Uri, Proxy-Scheme

Right.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 5 Oct 2013 16:28:29 +0200

On Oct 5, 2013, at 14:20, "Peter A. Bigot" <[email-masked]> wrote:

> However: DELETE with If-None-Match would return 4.12 Precondition Failed if the resource existed (as opposed to 2.05 Deleted or an authorization error if the conditional option was absent), and 4.04 Not Found if it didn't, so that actually already is a sort of HEAD low-cost probe for existence without returning the representation.  

And the same thing would work with GET as well, no?
The server could actually put an ETag on a 4.12 (ETag is "selected representation" metadata, 5.5.3), providing some more of the HEAD-style functionality Thomas was looking for.

> (Unless the server may check for authorization data first and return 4.03 Forbidden even if the resource doesn't exist, which case you can't tell anything from the result.)

Yes, authorization checks always need to go first so you don't disclose information about the presence of the resource to an unauthorized client.

Grüße, Carsten

---

From: "Peter A. Bigot" <[email-masked]>
Date: Sat, 05 Oct 2013 10:56:54 -0500

On 10/05/2013 09:28 AM, Carsten Bormann wrote:
>> However: DELETE with If-None-Match would return 4.12 Precondition
>> Failed if the resource existed (as opposed to 2.05 Deleted or an
>> authorization error if the conditional option was absent), and 4.04
>> Not Found if it didn't, so that actually already is a sort of HEAD
>> low-cost probe for existence without returning the representation.
>

Yes, and would be a safer approach since it's safe and idempotent.  So 
there is no reason for If-None-Match on DELETE.

> The server could
> actually put an ETag on a 4.12 (ETag is "selected representation"
> metadata, 5.5.3), providing some more of the HEAD-style functionality
> Thomas was looking for.

Yes, though it wouldn't be useful unless the client already knew what 
the ETag represented.

Huh.  If a resource is destroyed, and then a new resource created with 
the same URI, does the server have to make any given entity tag is not 
associated with a different resource representation for the new 
resource?  Probably so.

>
>> (Unless the server may check for authorization data first and
>> return 4.03 Forbidden even if the resource doesn't exist, which
>> case you can't tell anything from the result.)
>
> Yes, authorization checks always need to go first so you don't
> disclose information about the presence of the resource to an
> unauthorized client.

Good point regarding best-practices, but that's not in the 
specification.  It also needs clarification, since authorization may be 
relevant only for specific methods on specific resources.  Unless you 
stop using 4.04 entirely, the fact you got a 4.03 instead of a 4.04 
tells you there's something that could be there.

Peter

---

From: "Peter A. Bigot" <[email-masked]>
Date: Sat, 05 Oct 2013 10:57:40 -0500

Again the quote from 5.4 which is the heart of this matter:

On 10/05/2013 09:20 AM, Carsten Bormann wrote:
>> mentioned in other request methods. I don't think they make sense
>> for GET; I can imagine a use for POST; and If-Match (but not
>> If-None-Match) certainly makes sense for DELETE. Disallow for GET,
>> allow all others except If-None-Match for DELETE is disallowed.
>

Indeed.  5.4 is what mandates "exclusio alterius".  5.8.3 is one of the 
few sections that "expressio"d something.

>
> 5.10.8 makes no attempt at restricting these options to specific
> methods. (5.10.8.2 mentions some combinations are "not very useful",
> but doesn't attempt to rule them out.)

If they are not defined as possible options, 5.4 rules them out.

It would perhaps have been better if 5.4 had not done that, but it did.

I understand your position from the perspective of what was intended, 
and it's consistent with my proposed "interpretation".  But I don't 
understand your response from the perspective of what the specification 
requires.  5.10.8 doesn't have to restrict something in the specific 
case when it's already been restricted in the general case.

>> implicitly allowed for 0.02 POST requests and 2.xx Success
>> responses except for 2.03 ValidContent because these codes mention
>> a representation in the payload and 5.5.1 says Content-Format
>> applies to resource and action result payloads. 5.5.2 implies it
>> must not appear for a diagnostic payload, therefore cannot appear
>> in 4.xx or 5.xx error responses.
>
> 5.5.1 says that whenever the payload is a representation, the
> Content-Format option gives you its media type (and that applies to
> all methods and response codes).  It then weasels around the ability
> to leave off the option when the application can divine its value.
> The end of 5.5.1 combined with 5.5.2 says that if it is left off with
> a 4.xx/5.xx response, any payload present is not a representation but
> a diagnostic payload.  So there is no way to have a "format divined
> by application" representation with 4.xx/5.xx, but you can very well
> send a representation with these response codes, which is indicated
> by giving a Content-Format option.

Section 5.5:

    Both requests and responses may include a payload, depending on the
    method or response code respectively.  If a method or response code
    is not defined to have a payload, then a sender MUST NOT include one,
    and a recipient MUST ignore it.

None of the existing 4.xx/5.xx codes specify a payload, so none of them 
can have one, except under the general allowance in 5.9.2 and 5.9.3 that 
the server SHOULD include a diagnostic payload under certain circumstances.

So those codes can't have payloads other than diagnostic payloads, and 
diagnostic payloads can't have a Content-Format.  So Content-Format 
can't appear in responses with 4.xx/5.xx codes.  QED.

Yes, further specifications might change this for new result codes by 
defining resource representation and action result payloads.  That's 
irrelevant to what we're doing today.

>
> The diagnostic payload (read "not a representation") was invented as
> a way to transport the "reason phrase" of an HTTP status line where
> it most matters, which is in the error cases.
>
> I don't read 5.5.4 that way at all; it is actually quite useful to be
> able to select the content-format of a POST response. (Again, there
> is a helpful pointer in 5.8.1.)
>
>> But the other methods all have valid responses that can contain
>> non-diagnostic content, so maybe it's ok for them too?
>
> Yes.

Will that be in the errata for coap when the RFC comes out?

Because simply stating it here in the working group email list doesn't 
change the text of the RFC-to-be, or help future implementers.

Basically, 5.4 and 5.5 both put MUST restrictions on options and 
payloads, and the conditions on relaxing those restrictions were not met 
by the remainder of the text.  Too late now.  What's going to be done 
about it?

Peter

>
>> These options cannot appear (except as unrecognized) in any
>> request: Location-Path, Max-Age, Location-Query, Size1 (unless
>> block is supported)
>
> I don't think including a Size1 option in a request is outlawed by
> 5.10.9, it is just not very useful without -block, as it is redundant
> with the size of the actual payload supplied.
>
>> These options cannot appear (except as unrecognized) in any
>> response: If-Match, Uri-Host, If-None-Match, Uri-Port, Uri-Path,
>> Uri-Query, Accept, Proxy-Uri, Proxy-Scheme
>
> Right.
>
> Grüße, Carsten
>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 5 Oct 2013 18:16:04 +0200

On Oct 5, 2013, at 17:57, "Peter A. Bigot" <[email-masked]> wrote:

> Indeed.  5.4 is what mandates "exclusio alterius".  5.8.3 is one of the few sections that "expressio"d something.

I can see how you could read 5.4 like that.
But most options aren't even discussed in 5.8/5.9 at all.
Content-Format is only discussed for PUT in those sections.
If an option is defined for all of the codes, or for a class of them, there is no need to "define" it again in each method or response code; that would become very tedious.

Grüße, Carsten