Subject: [core] application/link-format & non CoRE applications

From: Herbert van de Sompel <[email-masked]>
Date: Tue, 13 Dec 2011 19:05:37 -0700

Hi all,

I am getting back to this list regarding the use of application/link-format
documents in the Memento "Time Travel for the Web" framework [1,2].

I have posted to this list before:

- To announce Memento's use of link-format documents [3].

- To point out that the determination of Context IRIs for links expressed
in application/link-format documents as specified in the CoRE Link Format
Internet Draft was problematic for Memento's use [4]. I never received any
reaction to this mail.

Having read the most recent version of the CoRE Link Format Internet Draft
[5] carefully, I need to conclude that the specification has evolved in
such a manner that use of the application/link-format media type in the
Memento framework has become quite problematic. And I expect it would be
problematic for many applications that are not CoRE.

I find this application-specific approach generally inappropriate for a
media type definition, and I find it especially cumbersome because I
anticipate that - as Link headers will become increasingly used - other
applications will encounter the need to express links in documents, not
just in HTTP headers. I think it would be counter-productive if such
applications would have to define their own media type based on the syntax
specified for the HTTP Link header [6]. Hence, I hope it will still be
possible to specify application/link-format in a more generic manner.

These are specific issues I would like to point out that make use of the
application/link-format media type problematic for applications other than
CoRE ones:

(a) Section 1 is entirely framed in terms of CoRE applications and states
"This document specifies a link format for use in CoRE Resource Discovery"

(b) Section 2 states "The CoRE Link Format is only expected to be supported
in constrained networks and M2M systems."

(c) As I expressed in [4], and as confirmed by the current version of the
I-D, the determination of the Context IRI for a link in
application/link-format documents is conceived in function of CoRE
applications and is problematic for other uses.  In Memento (and I
anticipate in other potential applications), the Context IRI is not the
base-URI of the Target URI nor the base-URI of the URI that delivered the
links. As a matter of fact, for the large majority of links used by
Memento, the Context IRI and Target IRI are in different domains. Hence,
following the specification in Section 2.1, all such links would have to be
expressed using an anchor:

- Doing so, as such, yields an enormous overhead (Memento TimeMaps that use
the link format can easily contain many hundreds of links; see e.g.
http://mementoarchive.lanl.gov/list/timemap/link/http://lanlsource.lanl.gov/hello
).

- To aggravate the situation, the wording regarding links with anchors in
the I-D marginalizes this approach by stating: "A consuming implementation
can however choose to ignore such links. It is not expected that all
implementations will be able to derive useful information from explicitly
anchored links." In [4], I described a proposal to allow specifying the
Context IRI of links in application/link-format documents by means of a
special-purpose link and a special-purpose relation type. I trust other
approaches can be imagined; the bottom line however is that a mechanism
that allows to elegantly deviate from the default determination of Context
IRIs as currently specified in the I-D should be available.

I would like to add that Memento's use of link-formatted documents is not
marginal. I anticipate that by the end of 2012 such documents will be
available to communicate the holdings of web archives, worldwide. That is
over 10 billion links expressed using this format.

Looking forward to a constructive discussion.

Greetings

Herbert Van de Sompel

[1] http://mementoweb.org

[2] https://datatracker.ietf.org/doc/draft-vandesompel-memento/

[3] http://www.ietf.org/mail-archive/web/core/current/msg01119.html

[4] http://www.ietf.org/mail-archive/web/core/current/msg01824.html

[5] http://tools.ietf.org/html/draft-ietf-core-link-format-09

[6] http://tools.ietf.org/html/rfc5988

-- 
Herbert Van de Sompel
Digital Library Research & Prototyping
Los Alamos National Laboratory, Research Library
http://public.lanl.gov/herbertv/

==

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 14 Dec 2011 08:49:54 +0100

[Trimming CC list -- yours is long enough to get the message stuck in the moderation queue.]

Herbert,

the CoRE WG has been working on using RFC 5988 Web Linking for its resource discovery needs.  The result is the link-format document that just passed second WGLC (the WGLC comments from this round still need to be processed).

What have we done here?
-- we have defined a way to ship link relations in a separate document that has its own media type (link-format).
-- we have defined the "hosts" link relation that is intended to be used for the "/.well-known/core" type application we are interested in.

Due to our application in constrained node/networks, we also are interested in minimizing the complexity that implementations need to go to.  So we allow some simplifications.  E.g., one WGLC comment was that expressing multiple link relations in a space-separated list is some complexity that is rarely needed in a CoRE application, so we are looking for ways to avoid it.  We will need to meet this kind of requirements to make link-format useful for its intended applications.

Note that the second-class status of anchored links wasn't invented by link-format, but is explicitly called out in 5988 as well (section 5.2, "Consuming implementations can choose to ignore links with an anchor parameter...").

What you are proposing amounts to re-using the link-format specification for a broader application.  Of course, most of us would agree that having multiple specifications for closely related purposes is rarely the right approach, so I'd expect many in the WG to support this.

So what is it that you need to re-purpose the link-format spec?

As far as I understand, the only actual change you need is less with link-format than with the context-IRI spec in 5988, which we aren't changing in link-format.
I'm a bit uneasy with specifying this change to the 5988 semantics (that we inherit) in the form of just another link relation -- the link relations in a link-format document are meant to be completely independent of each other; splitting a link-format document in the middle and conveying it as two documents should not change the semantics at all.

Maybe a different way is needed to influence the context-IRI of all links in a link-format document (i.e., beyond adding an anchor parameter per link).  I could even imagine applications of such a facility in CoRE, although in our space we would have to carefully weigh its usefulness against the added complexity.

Grüße, Carsten

---

From: Herbert van de Sompel <[email-masked]>
Date: Wed, 14 Dec 2011 13:52:51 -0700

Dear Carsten,

Many thanks for your response.

You are correct in stating that what we are looking for is the ability to
use the link format in applications other than CoRE. As I indicated, I
think it is a fair expectation for media types to be defined in a
relatively generic, not application-specific, manner. In the Memento
framework, link-formatted TimeMaps have been in use for more than a year,
and I suggest that is an indication of interest in the format beyond CoRE.

I do not agree with your perspective that our problem is with Context IRIs
as per RFC 5988. Rather it is with CoRE's approach to determine Context
IRIs for link-formatted documents:

1. In RFC 5988, the Context IRI of a link is by default that of the
responding URI. Because any URI can respond with a Link header, this allows
any URI to be Context IRI.

2. When defining the application/link-format media type for stand-alone
documents, CoRE significantly limits the kind of URIs that can function as
Context IRIs by:

2.1. Introducing the notion of base URI as Context IRI.

2.2. Deciding to marginalize anchored links.

3. In addition:

3.1. CoRE's default choice whereby the Context IRI is the base URI of
Target IRI is arbitrary, i.e. not based on RFC 5988.

3.2. The fallback choice whereby the Context IRI is the base URI of the
responding resource is not aligned with the logic of RFC 5988,  according
to which the responding URI itself would have to be the Context IRI.

4. Because, as you indicate, RFC 5988 _allows_ applications to marginalize
anchored links, and because CoRE decides to effectively marginalize them,
their use becomes really problematic in any application/link-format
document.

Based on the above, I conclude that what is needed to allow the
application/link-format media type to be used in applications beyond CoRE
is indeed what you describe as "a way to influence the Context IRI of all
links in a link-format document, different than adding anchors to links."
In addition to allowing any URI to be Context IRI, this ability would yield
self-containdeness of link-formatted documents: no information external to
a document would be needed in order to correctly interpret links. Note that
CoRE's approach (3.2) relies on external information, i.e. the base URI of
the responding URI.

My proposal to use a special-purpose link and link relation type was an
attempt at an approach to explicitly convey the Context IRI in
link-formatted documents. I agree with your argument that this makes the
interpretation of links dependent on another link; that is indeed not a
great idea and is at odds with RFC 5988 semantics. However, as I indicated
in my previous mail, other approaches can be imagined. Just as examples:

- The Context IRI could simply be stated in the first line of a
link-formatted document.

- A link-formatted document could use the HTTP header format using two HTTP
headers, i.e. Content-Location to express Context IRI and Link to express
links.

Greetings

Herbert Van de Sompel

On Wed, Dec 14, 2011 at 12:49 AM, Carsten Bormann <[email-masked]> wrote:

> the CoRE WG has been working on using RFC 5988 Web Linking for its
> resource discovery needs.  The result is the link-format document that just
> passed second WGLC (the WGLC comments from this round still need to be
> processed).
>
> What have we done here?
> -- we have defined a way to ship link relations in a separate document
> that has its own media type (link-format).
> -- we have defined the "hosts" link relation that is intended to be used
> for the "/.well-known/core" type application we are interested in.
>
> Due to our application in constrained node/networks, we also are
> interested in minimizing the complexity that implementations need to go to.
>  So we allow some simplifications.  E.g., one WGLC comment was that
> expressing multiple link relations in a space-separated list is some
> complexity that is rarely needed in a CoRE application, so we are looking
> for ways to avoid it.  We will need to meet this kind of requirements to
> make link-format useful for its intended applications.
>
> Note that the second-class status of anchored links wasn't invented by
> link-format, but is explicitly called out in 5988 as well (section 5.2,
> "Consuming implementations can choose to ignore links with an anchor
> parameter...").
>
> What you are proposing amounts to re-using the link-format specification
> for a broader application.  Of course, most of us would agree that having
> multiple specifications for closely related purposes is rarely the right
> approach, so I'd expect many in the WG to support this.
>
> As far as I understand, the only actual change you need is less with
> link-format than with the context-IRI spec in 5988, which we aren't
> changing in link-format.
> I'm a bit uneasy with specifying this change to the 5988 semantics (that
> we inherit) in the form of just another link relation -- the link relations
> in a link-format document are meant to be completely independent of each
> other; splitting a link-format document in the middle and conveying it as
> two documents should not change the semantics at all.
>
> Maybe a different way is needed to influence the context-IRI of all links
> in a link-format document (i.e., beyond adding an anchor parameter per
> link).  I could even imagine applications of such a facility in CoRE,
> although in our space we would have to carefully weigh its usefulness
> against the added complexity.
>
> Grüße, Carsten
>
>

==

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 14 Dec 2011 23:31:16 +0100

Herbert,

I definitely applaud you for trying not to re-invent the wheel.

To stay in the figure:
I'm just not so sure your truck will run well on our bicycle wheels.
(And I would like them to stay bicycle wheels.)
But then I don't know much about your truck.

<end of WG chair view influenced part of this message.>
Continuing on as an interested WG member:

> 1. In RFC 5988, the Context IRI of a link is by default that of the responding URI.

Indeed, we could have done the same thing, without "abbreviating away" the "/.well-known/core" part, at the cost of requiring anchors in almost everyone of our links (see below).

> Because any URI can respond with a Link header, this allows any URI to be Context IRI. 

Almost equivalently, you can put a link-format resource anywhere.
But I think you are objecting to that URI being subjected to some special treatment in link-format section 2.1.

> 2. When defining the application/link-format media type for stand-alone documents, CoRE significantly limits the kind of URIs that can function as Context IRIs by:

(I now think I understand -- you are objecting to the processing rules in link-format section 2.1 that lead to the value of the base-URI, not to the notion itself.)

> 2.2. Deciding to marginalize anchored links.

That might be the bicycle part.
(But then 5 of the 16 examples in the spec do use anchors!)

> 3. In addition:
> 3.1. CoRE's default choice whereby the Context IRI is the base URI of Target IRI is arbitrary, i.e. not based on RFC 5988. 

True.

> 3.2. The fallback choice whereby the Context IRI is the base URI of the responding resource is not aligned with the logic of RFC 5988,  according to which the responding URI itself would have to be the Context IRI. 

(Bicyles…)
True.
For relative-refs, this is the natural thing for the "hosts" relation, as the resource is hosted by the server, not by the links document.
For full URIs in the target-URI, our examples strangely all have an explicit anchor parameter.  But I think the rule is motivated by a resource directory talking about other servers.  (That kind of example should be added to the spec, I think.)

[…]
So how about making the same media format useful for both our trucks and bicycles:

> - The Context IRI could simply be stated in the first line of a link-formatted document.

We deliberately don't have the concept of lines so far, and I would like to avoid introducing them.

> - A link-formatted document could use the HTTP header format using two HTTP headers, i.e. Content-Location to express Context IRI and Link to express links.

This would need lines, and also add complexity for the simple case of a sensor node (note also that link-format is UTF-8 and not RFC2231-saddled).

Not being able to cover your objectives with the same media-type that we use does cause cognitive dissonance in this WG member, so I would be interested in other approaches.  (Note that a valid link-format always starts with a "<", so I can think about some simple syntactical hacks, e.g. just optionally prepending the context-IRI.  I'm not sure that I like what I can come up with in 10 minutes, though.)

The alternative would be to have two media types, one for trucks and one for bicycles.
They still could both be round (e.g., share the same ABNF and most of the processing rules).

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 15 Dec 2011 00:25:44 +0100

> For relative-refs, this is the natural thing for the "hosts" relation, as the resource is hosted by the server, not by the links document.

Hmm.  Food for thought: We could make the path-stripping semantics a property of the "hosts" relation.
(No, I haven't thought this through. Too late in the day for that.)

Can people please throw some recent CoRE link-format documents out of their real-life implementations at me/at the mailing list?
I'd like to make sure we actually do have the right fittings soldered to our bicycle wheels.  Thanks.

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Thu, 15 Dec 2011 20:53:14 +0200

> (No, I haven't thought this through. Too late in the day for that.)

Yes, the same thing came to mind for me when reading the TimeMap situation. We could very well narrow the scope of the base-URI to apply only to the "hosts" relation, that is sufficient for CoRE. 

However, that doesn't really solve their problem. The Context was the requested URI in RFC5988 as links are defined for use in the HTTP header, where obviously the Context can be something meaningful (a web page). But how RFC5988 defines the Context for the HTTP Link header, has no relevance to the context of links in a link document.  For the case of a link document, that obviously is not very useful, unless your only purpose is to describe relations between that link document and other resources (usecase?).  

So what would be the Context of other relations (other than hosts) for links in a link document? Obviously the Anchor attribute was meant for this purpose as defined in RFC5988, and there is nothing we do to prevent the use of them. We just point out the fact that constrained implementations will usually ignore them. I do agree that Anchors are inefficient, but at the same time we don't need to use Anchors very often in CoRE, so that is not a problem. I could understand that in a large link document repeating an anchor in every link is not so smart. 

As with Carsten, I am a bit hesitant to try to solve the problems of all possible link document applications in this WG. Of course we don't want to prevent people from re-using our work. If we can make a couple small changes so as not to prevent other uses, without affecting efficiency for CoRE, then I am happy to help.

Regarding the idea of specifying a "sticky anchor" that applies to all links in a document, I think that would be nice. But that mechanism probably should be described in some other document?

Zach 

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Herbert van de Sompel <[email-masked]>
Date: Thu, 15 Dec 2011 14:50:43 -0700

Dear Carsten, Zach,

Thanks for your supportive responses. I understand your use case and the
constraints it implies. As such I understand that the quest is for a small
change aimed at specifying a "sticky anchor" (thanks Zach!) in link-format
documents.

I discussed the problem with colleagues and we think there are two types of
solutions:

1. Approach that allows using an off-the-shelf Link header parser

This approach requires that the content of an application/link-format
document fully conforms to the "link-value" production rule of the ABNF of
RFC 5988. As a result, whatever is put in place to implement the "sticky
anchor" will be interpreted as a link.

My original proposal fits into this category, but probably was
inappropriate because the add-on I proposed looked too much like a "real"
link. Maybe it's better to go for something that conforms to the syntax of
a link but doesn't really make sense as a link, i.e. does not introduce the
risk of applications interpreting it as a "real" link.

Within this category, we thought the following could work:

<> ; anchor="the-actual-Context-IRI" , link, link, link, ...

2. Approach that requires some changes to an off-the-shelf Link header
parser

In this case, we could go beyond the syntax defined in the ABNF in order to
add the "sticky anchor". Devising a solution then comes down to finding a
verbose syntax that remains easy to parse. The latter includes deciding on
an appropriate delimiter. Understanding Carsten doesn't want "lines", maybe
using e.g. a pipe character "|" as a delimiter could be an approach:

| the-actual-Context-IRI | link, link, link, ...

or, if extensibility regarding metadata to these files would be required,
even:

| ContextIRI the-Context-IRI | Date the-date | link, link, link, ...

Ideas?

Greetings

Herbert

On Thu, Dec 15, 2011 at 11:53 AM, Zach Shelby <[email-masked]> wrote:

> >> For relative-refs, this is the natural thing for the "hosts" relation,
> as the resource is hosted by the server, not by the links document.
> >
> Yes, the same thing came to mind for me when reading the TimeMap
> situation. We could very well narrow the scope of the base-URI to apply
> only to the "hosts" relation, that is sufficient for CoRE.
>
> However, that doesn't really solve their problem. The Context was the
> requested URI in RFC5988 as links are defined for use in the HTTP header,
> where obviously the Context can be something meaningful (a web page). But
> how RFC5988 defines the Context for the HTTP Link header, has no relevance
> to the context of links in a link document.  For the case of a link
> document, that obviously is not very useful, unless your only purpose is to
> describe relations between that link document and other resources
> (usecase?).
>
> So what would be the Context of other relations (other than hosts) for
> links in a link document? Obviously the Anchor attribute was meant for this
> purpose as defined in RFC5988, and there is nothing we do to prevent the
> use of them. We just point out the fact that constrained implementations
> will usually ignore them. I do agree that Anchors are inefficient, but at
> the same time we don't need to use Anchors very often in CoRE, so that is
> not a problem. I could understand that in a large link document repeating
> an anchor in every link is not so smart.
>
> As with Carsten, I am a bit hesitant to try to solve the problems of all
> possible link document applications in this WG. Of course we don't want to
> prevent people from re-using our work. If we can make a couple small
> changes so as not to prevent other uses, without affecting efficiency for
> CoRE, then I am happy to help.
>
> Regarding the idea of specifying a "sticky anchor" that applies to all
> links in a document, I think that would be nice. But that mechanism
> probably should be described in some other document?
>
> Zach
>
> --
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

==