Subject: [Json] JSON encodings

From: Willem Bogaerts <[email-masked]>
Date: Wed, 30 Apr 2014 12:18:34 +0200

Dear Group,

I am new here, so this issue may have been handled in the past. I'd like
to solve an issue with JSON standard.

The issue is that according to RFC 7159, JSON is both text-based and
binary. And those two are not exactly compatible. With a mime-type of
"application/json", one would expect a binary format, which it really is
not. So one would expect a mime type like "text/json;charset=utf8", for
example. This would allow the parser to "get ready" for the data before
choking on it. In fact, most implementations check for an encoding, call
some library to convert it to an internally used encoding, and only then
call the JSON parser, which does not need to be aware of all the
possible encodings.

To do that, off course, the encoding should be separate from the data
itself. Not separating the two is like sending a locked safe with its
key inside: you will have to break open the safe to find the key that
opens it. In fact, some references to the JSON standard provide an
example on how to "crack the safe" by looking at the first 4 bytes.

This can never be the intention. I know that the HTML standard has a
history of "cracking" the pages for the http-header as a meta tag, but
this only serves as an example on how it should NOT be done.

To overcome the issue, I encountered content-types like
"application/json;charset=utf8". Although this seems (and is) a little
odd, I think this is the best way of preparing the client to the data it
is getting. Although JSON has an "application" base type, it really is
text and therefore needs an encoding.

So my comment to RFC 7159 is that it should be possible to send the
character encoding as meta-data when sending a JSON request. The absence
of this encoding then causes the request to be treated as utf-8.

In the section "IANA considerations", the standard now says "n/a" for
optional parameters to the mime type. I'd like to propose that the
"charset" parameter is an option.

This can also be a security consideration. There are illegal sequences
that can be misinterpreted and cause havoc. This is less likely with a
"pure data" format like JSON, but you never know where the ill-formed
data is going next. Somebody could try to inject characters that are not
recognised as quotes by a web page script, but are regarded as such by
the database that parses it as part of a query. You hardly need more for
a successful SQL injection attack. If I recall correctly, attacks like
this happened with UTF-7 encodings in the past.

Also, imagine that somebody follows the advice on looking at the first 4
bytes for the encoding and uses a forgiving encoding translator for the
rest. I am sure that it would be possible to start with, say, an
UTF-16LE encoded white space and add some nasty surprises in an other
encoding for the rest of the request. The encoding that is "cracked" out
of the first 4 bytes is not likely to be checked, as it was cracked out
of the request in the first place: there is nothing to validate it with.
Treating a piece of text as a binary makes this kind of attacks possible.

Stating the character encoding beforehand can prevent any surprises in
how the request is parsed, can serve as a meaningful validation, and
allows the programmer to send meaningful error messages for any
ill-encoded request.

Best regards,
-- 
Willem Bogaerts

Application Smith
Kratz Business Solutions
http://www.kratz.nl/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 30 Apr 2014 14:44:01 +0200

On 2014-04-30 12:18, Willem Bogaerts wrote:
> ...

Willem,

application/json doesn't have a charset parameter, and never had one.

We had long discussions about what encodings should be allowed, and how 
to detect them. I recommend that you have a look at the mailing list 
archives of the last ~6 months.

Best regards, Julian