Subject: [core] Proxy Meda Type Conversion

From: Guido Moritz <[email-masked]>
Date: Mon, 12 Dec 2011 17:02:21 +0100

Dear WG,

I have a question I cannot answer myself and would be interested in your
thoughts and comments. It is maybe somewhat strange, but might be important
in the sense of usage of CoAP in existing environments. As far as I know
there is currently no mechanism in CoAP to realize what I describe below. If
I am wrong, please forget about this mail and do more important things
(please can you give me a hint to the solution in this case anyway).

Let's assume an application in the unconstrained world which is using HTTP
and XML for the data representation. We don't want to change this
implementation completely and so we advise the CoAP/HTTP Proxy to do the
required conversions of CoAP and HTTP when communicating with the 6LoWPAN.
But there are optimized encodings existing also for the XML part. EXI is a
good candidate, but let's not limit to it. So we have several optimized
serialization formats for one and the same data, which can also be mapped
seamless, transparent and stateless in the proxy. Is there a mechanism in
the proxy spec to advise the proxy to (re-)encode also the payload? Of
course the endpoints have to advise the proxy explicitly, because the proxy
is stateless over different request. I was thinking about some weird usage
of the Accept-Option, but I didn't come to a comprehensive conclusion. 

Btw, please don't ask me about the specific application I have in mind. It
is (again) about usage of SOAP in 6LoWPANs and related to the SOAP-over-CoAP
idea. Hence, you will not like my answer to the question about the
application scenario anyway. :)

Regards,
Guido

---

From: TianLinyi <[email-masked]>
Date: Tue, 13 Dec 2011 05:20:17 +0000

Hi, Guido

It is not clear to me what the following question means.
"Is there a mechanism in the proxy spec to advise the proxy to (re-)encode also the payload?"

Do you mean that the proxy needs to parse the payload in Exi format to another format to be included in the coap message? There are some binary xml format, e.g. WBXML or EXI. The accept header indicates the MIME type expected by the sensor. When the proxy receives the message from outside, it has no idea what MIME type the sensors wants. This would only happen when proxy understand the service logic, but that would be too much for the proxy, right?

Cheers,
Linyi

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Guido Moritz
Sent: Tuesday, December 13, 2011 12:02 AM
To: 'core'
Subject: [core] Proxy Meda Type Conversion

Dear WG,

Regards,
Guido

_______________________________________________
core mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 13 Dec 2011 08:27:47 +0000

Hi Guido,

Indeed not specified at this moment. Of course one could go and build a custom, modified CoAP-HTTP proxy that implements the functionality you describe. Although we have no use case yet it's interesting since it allows CoAP nodes with binary xml support to communicate with 'legacy' Web services that do not understand any binary xml formats. This might be a topic for a future I-D ... ?

For CoAP requests translated by a CoAP-HTTP proxy to uncompressed XML inside a HTTP request I can see it clearly; for HTTP requests with an xml payload, sent via a HTTP-CoAP proxy to a CoAP end-point, I don't see clearly how the proxy would know to what format to convert (as mentioned by Linyi).

Esko

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of TianLinyi
Sent: Tuesday 13 December 2011 6:20
To: Guido Moritz; 'core'
Subject: Re: [core] Proxy Meda Type Conversion

Hi, Guido

Cheers,
Linyi

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Guido Moritz
Sent: Tuesday, December 13, 2011 12:02 AM
To: 'core'
Subject: [core] Proxy Meda Type Conversion

Dear WG,

Regards,
Guido

________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: Guido Moritz <[email-masked]>
Date: Tue, 13 Dec 2011 09:58:08 +0100

You got exactly the point! We have the plan to implement such a proxy. So we
already have an EXI implementation for TelosB and we have a HTTP-CoAP and
CoAP-HTTP Proxy. Now the idea was to also implement a conversion of XML to
EXI and vice versa in the Proxy. 

Off course the proxy is stateless and has no knowledge about which
conversion has to be done for which endpoints. Hence, this information must
be somehow encoded in the request and response. How the endpoints receive
the knowledge about which formats are supported by the other side is out of
scope so far. This could be maybe performed during the discovery phase,
static defined, etc. For our use case it is sufficient to perform the
conversion statically. 

The question was, if such a mechanism to advise the proxy to also do the
data conversion already exists in CoAP. I was not sure, but it seems not to
be the case. Putting this issue (maybe including our use case) in an I-D
seems reasonable...maybe I will find some time during Christmas days. 

Nevertheless, more comments and ideas on that issue would be great!

Best,
Guido

> -----Ursprüngliche Nachricht-----
> Von: Dijk, Esko [mailto:[email-masked]]
> Gesendet: Dienstag, 13. Dezember 2011 09:28
> An: TianLinyi; Guido Moritz
> Cc: 'core'
> Betreff: RE: [core] Proxy Meda Type Conversion
> 
> Hi Guido,
> 
> Indeed not specified at this moment. Of course one could go and build a
> custom, modified CoAP-HTTP proxy that implements the functionality you
> describe. Although we have no use case yet it's interesting since it
allows CoAP
> nodes with binary xml support to communicate with 'legacy' Web services
that
> do not understand any binary xml formats. This might be a topic for a
future I-D
> ... ?
> 
> For CoAP requests translated by a CoAP-HTTP proxy to uncompressed XML
> inside a HTTP request I can see it clearly; for HTTP requests with an xml
> payload, sent via a HTTP-CoAP proxy to a CoAP end-point, I don't see
clearly
> how the proxy would know to what format to convert (as mentioned by
Linyi).
> "Is there a mechanism in the proxy spec to advise the proxy to (re-)encode
also
> the payload?"
> 
> Do you mean that the proxy needs to parse the payload in Exi format to
> another format to be included in the coap message? There are some binary
> xml format, e.g. WBXML or EXI. The accept header indicates the MIME type
> expected by the sensor. When the proxy receives the message from outside,
it
> has no idea what MIME type the sensors wants. This would only happen when
> proxy understand the service logic, but that would be too much for the
proxy,
> right?
> The information contained in this message may be confidential and legally
> protected under applicable law. The message is intended solely for the
> addressee(s). If you are not the intended recipient, you are hereby
notified that
> any use, forwarding, dissemination, or reproduction of this message is
strictly
> prohibited and may be unlawful. If you are not the intended recipient,
please
> contact the sender by return e-mail and destroy all copies of the original
> message.

---

From: Thomas Fossati <[email-masked]>
Date: Tue, 13 Dec 2011 10:52:00 +0100

Hi Guido,

> Nevertheless, more comments and ideas on that issue would be great!

I may have misunderstood your point, but *provided the existence of a schema-less translation of XML to EXI* (and viceversa), couldn't the intermediate node simply do "proxy" content negotiation, like this ?

HTTP UA   Proxy    CoAP srv
  |         |         |
  +-------->|         | Accept: text/xml, application/xml 
  |  GET x  |         |
  |         +-------->| Accept: application/xml, application/exi
  |         |  GET y  |
  |         |         |

and viceversa:

CoAP cli  Proxy    HTTP origin
  |         |         |
  +-------->|         | Accept: application/exi
  |  GET a  |         |
  |         +-------->| Accept: application/xml, text/xml, Accept-Encoding: x-exi
  |         |  GET b  |
  |         |         |

handling the XML<->EXI conversions transparently.

[The above scenario comprises an HTTP UA and origin who can do XML (the origin may perhaps do EXI, see Accept-Encoding) and a CoAP client and server who can only handle EXI encoded XML.]

---

From: Guido Moritz <[email-masked]>
Date: Tue, 13 Dec 2011 11:17:42 +0100

Thomas,

the problem is that in the case you described only the response can be
optimized, as the proxy can indicate supported formats in the Accept Option.
But there is currently no way to do it also for the request. The only way to
it this is do always the conversion (which is so far OK for our use case,
but for sure not for all other use cases too).

The scenario maybe explained more precisely: A XML over HTTP client knows
about a sensor to invoke and knows that the sensor supports EXI over CoAP
(maybe also XML over CoAP, but this is somewhat optimization already). So
for the request it uses a HTTP-CoAP proxy. But the proxy has no knowledge
about the supported formats of the sensor. So the HTTP client would like to
instruct the proxy also to do the format conversion.

Let's turn this around: A EXI over CoAP client wants to deliver the data to
an XML over HTTP server. So it uses the CoAP-HTTP proxy. But the proxy has
no knowledge about the supported formats of the server. So the CoAP client
would like to instruct the proxy also to do the format conversion.

Such a mechanism would be quite useful to integrate CoAP sensors in existing
applications, because no (not too many) changes have to be done in the
existing implementations. The only new parts are the sensors and the proxy
including EXI/XML (and maybe more formats). And the proxy is still
stateless, because it requires no knowledge about supported formats of each
single endpoint.

Guido

> Von: Thomas Fossati [mailto:[email-masked]]
> Gesendet: Dienstag, 13. Dezember 2011 10:52
> An: Guido Moritz
> Cc: [email-masked] WG
> Betreff: Re: [core] Proxy Meda Type Conversion
> 
> Hi Guido,
> 
> I may have misunderstood your point, but *provided the existence of a
schema-
> less translation of XML to EXI* (and viceversa), couldn't the intermediate
node
> simply do "proxy" content negotiation, like this ?
> 
> HTTP UA   Proxy    CoAP srv
>   |         |         |
> [The above scenario comprises an HTTP UA and origin who can do XML (the
> origin may perhaps do EXI, see Accept-Encoding) and a CoAP client and
server
> who can only handle EXI encoded XML.]=

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Tue, 13 Dec 2011 11:30:48 +0100

Web proxies are not intended to convert application data, so probably
this feature is missing for this reason.

You can get around it as Thomas suggested to you, and you can even
instantiate a long-lived observe session with that Accept options.. In
this way your ALG/proxy will also convert the payload even on push
operations from nodes by remembering the state for that conversion.

Otherwise we will need a special option for CoAP requests to specify
application data conversion for an intermediate proxy, and handle all
the complexities this leads to... I doubt that this will be done in
the WG, anyway I personally think that it may be useful because I may
have similar needs too.

Are there other options to get around this?

Best,
Angelo

---

From: Thomas Fossati <[email-masked]>
Date: Tue, 13 Dec 2011 11:47:50 +0100

Hi Guido,

> but for sure not for all other use cases too).

ok, or otherwise performing offline/online discovery of the resources hosted by the target node, and cache its links and associated ct's.
This needs a cooperative client of course, and an "intelligent" proxy too.

> instruct the proxy also to do the format conversion.

If the HTTP/XML UA knows in advance the specific encoding accepted by the sensor, why couldn't it handle the correct encoding by itself ?

> would like to instruct the proxy also to do the format conversion.

Ok, this could be more problematic than the previous case because we are on the constrained side of the communication path.  Here a "Proxy-Transform" option could make sense.

> stateless, because it requires no knowledge about supported formats of each

A cacheing proxy wouldn't be stateless anyway, but I agree with you that translating-only intermediaries could benefit from such mechanism.

---

From: Guido Moritz <[email-masked]>
Date: Tue, 13 Dec 2011 11:50:25 +0100

I agree with your point that application data conversion for an intermediate proxy cannot be specified in CoRE. This would be in the responsibility of other groups or even other standardization bodies. 

But it would be the WG (if the WG is interested in) to provide and define a protocol mechanism, which can be used for such purposes. Maybe at the end this is only an option with an arbitrary data type, which can be used by other bodies to express the type and way of conversion.

If there is an existing solution for this I would be happy with it anyway. 

Best,
Guido

> Von: [email-masked] [mailto:[email-masked]] Im
> Auftrag von Angelo P. Castellani
> Gesendet: Dienstag, 13. Dezember 2011 11:31
> An: Guido Moritz
> Cc: Thomas Fossati; [email-masked]
> >> > CoAP-HTTP Proxy. Now the idea was to also implement a conversion of
> XML
> >> to

---

From: Guido Moritz <[email-masked]>
Date: Tue, 13 Dec 2011 11:58:46 +0100

> If the HTTP/XML UA knows in advance the specific encoding accepted by the
> sensor, why couldn't it handle the correct encoding by itself ?

This is more or less a question about the application. Indeed, this is to
some extend only useful if one does not want or can change the unconstrained
part (HTTP, XML). So maybe I can answer with a question: If the HTTP/XML UA
knows in advance that the sensor only features CoAP and not HTTP, why does
it not uses CoAP directly but uses the proxy at all?

After all the layers that can be compressed an optimized now (6LoWPAN, UDP,
CoAP...), a possible stateless conversion also for the payload is from my
point of view only the next (maybe the final) step to be done. 

Guido

> Gesendet: Dienstag, 13. Dezember 2011 11:48
> An: Guido Moritz
> Cc: [email-masked] WG
> Betreff: Re: AW: [core] Proxy Meda Type Conversion
> 
> Hi Guido,
> 
> ok, or otherwise performing offline/online discovery of the resources
hosted by
> the target node, and cache its links and associated ct's.
> If the HTTP/XML UA knows in advance the specific encoding accepted by the
> sensor, why couldn't it handle the correct encoding by itself ?
> 
> Ok, this could be more problematic than the previous case because we are
on
> the constrained side of the communication path.  Here a "Proxy-Transform"
> option could make sense.
> 
> A cacheing proxy wouldn't be stateless anyway, but I agree with you that
> translating-only intermediaries could benefit from such mechanism.

---

From: Thomas Fossati <[email-masked]>
Date: Tue, 13 Dec 2011 12:01:34 +0100

> I agree with your point that application data conversion for an intermediate proxy cannot be specified in CoRE. This would be in the responsibility of other groups or even other standardization bod…

Yes, and besides, being on the unconstrained side must have some downsides too :-)

But seriously, if the HTTP client knows which format the sensor is able to consume, I think it is its responsibility to handle the correct encoding.

> But it would be the WG (if the WG is interested in) to provide and define a protocol mechanism, which can be used for such purposes. Maybe at the end this is only an option with an arbitrary data t…

I'm in favour of a new "Proxy-Translate", or similar, option.

> If there is an existing solution for this I would be happy with it anyway. 

No currently defined option seems able to cleanly do the trick.

---

From: pierpaolo giacomin <[email-masked]>
Date: Tue, 13 Dec 2011 13:25:20 +0200

I was discussing yesterday with tho of handling application data 
conversion in the proxy. The objective of my proposal was having a 
coalesced cache for different content-types.
This would violate heavily the concept of end-to-end, but still, it 
presents interesting pros, namely reducing communication between proxy 
and the cached resource, and memory usage in the proxy.
Anyway, yesterday he convinced me to drop it, with a strong argument 
about origin preservation.
Moreover, I've not enough background on xml, exi and json to state a 
possible translation 1:1 exists, or at least from a given format to two 
the others. If someone does I'll be happy to pursue the idea, and that 
could fix as well your needs.
If a 1:1 translation is not available it would become definitely an 
application issue, moving away from "straight-proxy" scope. Still 
interesting to me, but no point to speak about it here.

On 13.12.2011 12:50, Guido Moritz wrote:
> I agree with your point that application data conversion for an
> intermediate proxy cannot be specified in CoRE. This would be in the
> responsibility of other groups or even other standardization bodies.
>
> But it would be the WG (if the WG is interested in) to provide and
> define a protocol mechanism, which can be used for such purposes.
> Maybe at the end this is only an option with an arbitrary data type,
> which can be used by other bodies to express the type and way of
> conversion.
>
>> On Tue, Dec 13, 2011 at 11:17, Guido Moritz 
>> <[email-masked]>
>> wrote:
>> > Thomas,
>> >
>> >> > You got exactly the point! We have the plan to implement such a 
>> proxy.
>> > So we
>> >> > already have an EXI implementation for TelosB and we have a 
>> HTTP-CoAP
>> >> and
>> >> > conversion has to be done for which endpoints. Hence, this 
>> information
>> > must
>> >> > be somehow encoded in the request and response. How the 
>> endpoints
>> > receive
>> >> > the knowledge about which formats are supported by the other 
>> side is out
>> > of
>> >> > data conversion already exists in CoAP. I was not sure, but it 
>> seems not
>> > to

---

From: pierpaolo giacomin <[email-masked]>
Date: Tue, 13 Dec 2011 13:36:13 +0200

-1
I'm against this "Proxy-Translate" option.
The proxy receive in the "Accept" the palatable formats, the proxy 
should be aware which formats supported by the resource it is able to 
translate into the client's palatable ones.

On 13.12.2011 13:01, Thomas Fossati wrote:
> But seriously, if the HTTP client knows which format the sensor is
> able to consume, I think it is its responsibility to handle the
> correct encoding.
>

---

From: Guido Moritz <[email-masked]>
Date: Tue, 13 Dec 2011 12:43:04 +0100

> the others. 
There are formats exist which can be converted into each other without any state. For SOAP and thus XML-Infoset based documents the first which come into my mind are native XML, EXI and Fast Infoset. 

> interesting to me, but no point to speak about it here.
Absolutely agree!

Guido

> Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
> pierpaolo giacomin
> Gesendet: Dienstag, 13. Dezember 2011 12:25
> An: [email-masked]

---

From: Thomas Fossati <[email-masked]>
Date: Tue, 13 Dec 2011 14:02:39 +0100

> If there is an existing solution for this I would be happy with it anyway. 

One further element.  HTTP doesn't provide any way for the HTTP/CoAP proxy to handle this situation transparently, see:
 
  http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-17#section-7.4.16

On a 415 response, the origin would not be forced to express any hint about the supported media types to the failed proxy request.

So, a transparent conversion handled by the proxy using a trial and error strategy is not applicable in the general case, and a translation hint provided by the CoAP client via option seems to fix this hole nicely, IMHO.