Subject: Re: [IPP] What natural-language registry is used by IPP?

From: Ira McDonald <[email-masked]>
Date: Tue, 1 May 2012 11:25:09 -0400

Hi,

Per the IETF RFC Index, the latest successors to RFC 1766 are:

5646 Tags for Identifying Languages. A. Phillips, Ed., M. Davis, Ed..
     September 2009. (Format: TXT=208592 bytes) (Obsoletes RFC4646) (Also
     BCP0047) (Status: BEST CURRENT PRACTICE)

4647 Matching of Language Tags. A. Phillips, M. Davis. September 2006.
     (Format: TXT=45595 bytes) (Obsoletes RFC3066) (Also BCP0047) (Status:
     BEST CURRENT PRACTICE)

Language tags can now include embedded tags for scripts,
geographic regions (not just countries), variants, and private
use tags.

Parsing language tags is no longer simple.  Beware that they
embedded the script tag (if present) BEFORE the region tag
(e.g., 'US').  Naive parsers will fail badly.  See the ABNF on
pages 5-6 of RFC 5646.  Reference libraries for parsing do
exist.

Both simple language tags (e.g., 'en') and simple region tags
(e.g., 'US') can now be EITHER 2 or 3 characters.  Script
tags must be exactly 4 characters.  Variant tags can be 5-8
characters.  See the examples right after the ABNF.

Of course the fixed language and country tags of 2 characters
in the Printer MIB v2 are broken - they were broken when it
was published, but I failed to convince the other editors to let
me fix it (many naive clients would have been broken).

BEWARE:  3-character language tags are in common use
and are in fact preferred where an older 2-character tag was
already registered (I can't explain why - never understood it).

Cheers,
- Ira

Ira McDonald (Musician / Software Architect)
Chair - Linux Foundation Open Printing WG
Secretary - IEEE-ISTO Printer Working Group
Co-Chair - IEEE-ISTO PWG IPP WG
Co-Chair - TCG Trusted Mobility Solutions WG
Chair - TCG Embedded Systems Hardcopy SG
IETF Designated Expert - IPP & Printer MIB
Blue Roof Music/High North Inc
http://sites.google.com/site/blueroofmusic
http://sites.google.com/site/highnorthinc
mailto:[email-masked]
Winter  579 Park Place  Saline, MI  48176  734-944-0094
Summer  PO Box 221  Grand Marais, MI 49839  906-494-2434

On Mon, Apr 30, 2012 at 8:04 PM, Michael Sweet <[email-masked]> wrote:

> Glen,
>
> On Apr 30, 2012, at 3:59 PM, Petrie, Glen wrote:
>
> Can someone provide me the reference for the natural-language registry is
> used by IPP?****
>
>
> From RFC 2911:
>
> 4.1.8 'naturalLanguage'
>
>    The 'naturalLanguage' attribute syntax is a standard identifier for a
>    natural language and optionally a country.  The values for this
>    syntax type are defined by RFC 1766 <http://tools.ietf.org/html/rfc1766> [RFC1766 <http://tools.ietf.org/html/rfc1766>].  Though RFC 1766 <http://tools.ietf.org/html/rfc1766>
>    requires that the values be case-insensitive US-ASCII [ASCII <http://tools.ietf.org/html/rfc2911#ref-ASCII>], IPP
>    requires all lower case to simplify comparing by IPP clients and
>    Printer objects.  Examples include:
>
>       'en':  for English
>       'en-us': for US English
>       'fr': for French
>       'de':  for German
>
>    The maximum length of 'naturalLanguage' values used to represent IPP
>    attribute values is 63 octets.
>
>
> which leads to RFC 3282 (the replacement), which references ISO 639,
> 639-2, 3166, and 15924.
>
> _________________________________________________________
> Michael Sweet, Senior Printing System Engineer, PWG Chair
>
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
>
> _______________________________________________
> ipp mailing list
> [email-masked]
> https://www.pwg.org/mailman/listinfo/ipp
>
>

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
ipp mailing list
[email-masked]
https://www.pwg.org/mailman/listinfo/ipp

---

From: Michael Sweet <[email-masked]>
Date: Tue, 01 May 2012 16:00:40 -0700

Ira,

There appears to be a serious issue with how the IETF is tracking 1766. The following link points to 3066 AND 3282:

    http://tools.ietf.org/html/rfc1766

3282 has not been updated.

3066 points to 4646 and 4647, and 4646 points to 5646.

So the currently-in-effect RFCs (replacing 1766) are:

    RFC 3282: Content Language Headers
    RFC 4647: Matching of Language Tags
    RFC 5646: Tags for Identifying Languages

I am not going to rant about what the ISO and IETF have done to language tags. However, I do think it would be useful to add some text to IPP Everywhere to talk about the current state of affairs and some general guidelines, namely:

1. Use the rules in RFC 4647 for matching language tags.

2. Validate language tag values by supported characters - lowercase letters, digits, and "-".

3. Do not support extended ISO 639 language codes since technically they violate RFC 5656 (which requires the shortest ISO 639 code be used); for example, "english" is not OK but "en", "en-us", "en-uk", etc. are fine.

4. Expect Clients to make requests with language codes that do not match the generated-natural-language-supported values but map to them using the matching rules in #1.

For #4 the common issues I've dealt with in CUPS are:

- "no" (formerly "Norwegian Bokmål) is now "nb" for clarity; the other (less) common language in Norway is "Norwegian Nynorsk" ("nn").
- "zh-cn" is now "zh-hans", "zh-hans-cn", or "zh-hant-cn" (based on usage on Linux/UNIX)
- "zh-tw" is now "zh-hant", "zh-hans-tw", or "zh-hant-tw" (based on usage on Linux/UNIX)

The latter issue is the one that causes the greatest interoperability issues, and one that I have fielded many bugs for in CUPS.  In some future release libcups will have a secret decoder ring API that does matching and substitution in the face of two different language codes that mean the same thing.

Thoughts?

On May 1, 2012, at 8:25 AM, Ira McDonald wrote:

>> Can someone provide me the reference for the natural-language registry is used by IPP?
> 
> 
> From RFC 2911:
> 
> 
> 4.1.8 'naturalLanguage'
> 
>    syntax type are defined by RFC 1766 [RFC1766].  Though RFC 1766
>    requires that the values be case-insensitive US-ASCII [ASCII], IPP
> which leads to RFC 3282 (the replacement), which references ISO 639, 639-2, 3166, and 15924.
>

---

From: "Petrie, Glen" <[email-masked]>
Date: Fri, 4 May 2012 09:56:21 -0700

Mike 

<snip....>

________________________________

I am not going to rant about what the ISO and IETF have done to language
tags. However, I do think it would be useful to add some text to IPP
Everywhere to talk about the current state of affairs and some general
guidelines, namely:

[gwp] I believe it would also be useful.  After investigating languages
tags over the last few days; it can be quite complex since only do you
have the ISO (639-X) tags but Microsoft has a list, XML has a list,
....................... it appears that everyone has their own list.
While most list overlap, it still looks a ""little"" messy.   I believe
this discussion is important to both IPP AND to Cloud. 

<... snip ...> 

3. Do not support extended ISO 639 language codes since technically they
violate RFC 5656 (which requires the shortest ISO 639 code be used); for
example, "english" is not OK but "en", "en-us", "en-uk", etc. are fine

[gwp] I did not understand that ISO 639 (-3) was using "english" as a
tag but was using "eng"; I understand that the new ISO 639 tags are
three chars in length and I have not discovered extensions (eng-xxx);
yet!  Note that ISO 639-3 has some 8000+ tags; so that some "secret
decoder ring".

[gwp] Question: Should PWG create it own list of language tags?  The
list could be based on what is generally used by everyone else but not
all of the ones in ISO 639-3.  I do not know if a language tag hierarchy
exists that would reduce the ISO 639-3 list down to more manageable list
of tags.

glen

---

From: Michael Sweet <[email-masked]>
Date: Fri, 04 May 2012 10:09:08 -0700

Glen,

On May 4, 2012, at 9:56 AM, "Petrie, Glen" <[email-masked]> wrote:
> ... 
> 3. Do not support extended ISO 639 language codes since technically they violate RFC 5656 (which requires the shortest ISO 639 code be used); for example, "english" is not OK but "en", "en-us", "en-uk", etc. are fine
>  
> [gwp] I did not understand that ISO 639 (-3) was using “english" as a tag but was using “eng”; I understand that the new ISO 639 tags are three chars in length and I have not discovered extensions (eng-xxx); yet!  Note that ISO 639-3 has some 8000+ tags; so that some “secret decoder ring”.

My example wasn't necessarily the best; will update accordingly.

As for the decoder ring, I'm fine keeping the list short to those people are actually using widely (which keeps us to around 50 or so...)

> [gwp] Question: Should PWG create it own list of language tags?  The list could be based on what is generally used by everyone else but not all of the ones in ISO 639-3.  I do not know if a language tag hierarchy exists that would reduce the ISO 639-3 list down to more manageable list of tags.

I would not want the PWG to tackle defining its own registry - that way lies madness... :)

IANA already has a registry that we can (and should) use.  However I am not opposed to providing an example list of common language tags and aliases, with a pointer to the IANA registry as the definitive source.  Ideally this should be part of an implementer's guide, but since I don't think we want to update RFC 3196 it can go in a non-normative appendix that references RFC 3196 as part of IPP Everywhere.

__________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: "Petrie, Glen" <[email-masked]>
Date: Fri, 4 May 2012 10:31:40 -0700

Mike

And yes, it is a mess... :(

[gwp] No kidding; this looks worse and bigger than ISO 639-3!!!!

Again thanks.

glen

From: Michael Sweet [mailto:[email-masked]] 
Sent: Friday, May 04, 2012 10:20 AM
To: Petrie, Glen
Subject: Re: [IPP] What natural-language registry is used by IPP?

Glen,

The IANA language subtag registry is here:

            http://www.iana.org/assignments/language-subtag-registry

 

On May 4, 2012, at 10:15 AM, "Petrie, Glen" <[email-masked]>
wrote:

Mike,

 

IANA already has a registry that we can (and should) use.  However I am
not opposed to providing an example list of common language tags and
aliases, with a pointer to the IANA registry as the definitive source.
Ideally this should be part of an implementer's guide, but since I don't
think we want to update RFC 3196 it can go in a non-normative appendix
that references RFC 3196 as part of IPP Everywhere.

[gwp] What is the name of the IANA registry?   I can find stuff on
"format" but not a list.

Thanks ahead of time, glen

---

From: "Petrie, Glen" <[email-masked]>
Date: Fri, 4 May 2012 13:46:19 -0700

Mike

<...snip...>

I would not want the PWG to tackle defining its own registry - that way
lies madness... :)

[gwp] what I really meant was something like MSN.   I would hope that
most major languages, like media sizes, are "established"; so it is a
matter of defining a set appropriate for print.

Glen

---

From: Ira McDonald <[email-masked]>
Date: Sun, 6 May 2012 16:33:17 -0400

Hi,

I agree w/ Mike about an Informative appendix in IPP Everywhere
that updates IPP/1.1 Implementor's Guide (RFC 3196) with a good
discussion of matching of (underspecified and overspecified) tags
for natural languages.

Glen - note that the IANA subtag registry was machine-generated
by Doug Ewell in collaboration with the actual ISO standard groups
that produced all the source tags - this registry was verified by folks
from Unicode Consortium, Microsoft, Google, and others active in
the IETF LTRU WG - I was active for ten years in this IETF WG.

I would not characterize that IANA registry as "a mess" - it's used in
the message catalog and text processing software in a lot of operating
systems and all major web browsers.

Tricky - yes - that's the point of lifting examples from RFC 4647 to
show the language tag matching rules in practice.

The list for printers of 50 common natural languages is useless - I've
seen plenty of printers that support 100 or more.

But document DATA can be in any language (not just the console ones
in IPP generated-natural-language-supported) and UTF-8 or other
Unicode transcoding content can be correctly rendered using the
correct glyphs for the script and language by printers from many
manufacturers.

Cheers,
- Ira

On Fri, May 4, 2012 at 1:31 PM, Petrie, Glen <[email-masked]>wrote:

>  Mike****
>
> ** **
>
> And yes, it is a mess... :(****
>
> ** **
>
> [gwp] No kidding; this looks worse and bigger than ISO 639-3!!!!****
>
> ** **
>
> Again thanks.****
>
> ** **
>
> glen****
>
> ** **
>  ------------------------------
>
> *From:* Michael Sweet [mailto:[email-masked]]
> *Sent:* Friday, May 04, 2012 10:20 AM
> *To:* Petrie, Glen
> *Subject:* Re: [IPP] What natural-language registry is used by IPP?****
>
> ** **
>
> Glen,****
>
> ** **
>
> The IANA language subtag registry is here:****
>
> ** **
>
>             http://www.iana.org/assignments/language-subtag-registry****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> wrote:****
> ****
> Mike,********
>  ****
>  ****
>   ------------------------------
>
>  ****
>
>  ****
>
> think we want to update RFC 3196 it can go in a non-normative appendix that
> references RFC 3196 as part of IPP Everywhere.********
>
>  ****
>
> “format” but not a list.********
>
>  ****
>
> Thanks ahead of time, glen********
>
> ** **
>
> __________________________________________________****
>
> Michael Sweet, Senior Printing System Engineer, PWG Chair****
>
> ** **
>
> --