Subject: Re: [RTG-DIR] RtgDir review: draft-rtg-dt-encap-02

From: Erik Nordmark <[email-masked]>
Date: Mon, 6 Jul 2015 19:17:20 +0200

On 6/30/15 2:20 PM, Jamal Hadi Salim wrote:
> I have been selected as the Routing Directorate reviewer for this draft.
> The Routing Directorate seeks to review all routing or routing-related
> drafts as they pass through IETF last call and IESG review, and sometimes
> on special request. The purpose of the review is to provide assistance to
> the Routing ADs. For more information about the Routing Directorate, 
> please see http://trac.tools.ietf.org/area/rtg/trac/wiki/RtgDir
>
> Although these comments are primarily for the use of the Routing ADs, it
> would be helpful if you could consider them along with any other IETF
> Last Call comments that you receive, and strive to resolve
> them through discussion or by updating the draft.
>
> Document: draft-rtg-dt-encap-02
> Reviewer: Jamal Hadi Salim
> Review Date: 6/30/15 (later than requested, sorry)
> Intended Status: Informational
> WG LC End Date: unknown
>
> Summary:
>
> The document has significant good work and recommendations for
> encapsulation design. Many years of experience in issues found
> with encapsulation deployments are discussed. There are times
> where i lost track what the document was about because issues
> were being discussed without making recommendations on what is needed
> from an encapsulation perspective to deal with those issues; otoh,
> a good read is section 18 which would mention an issue and in the
> same breath suggests how a design should handle said issue.
Jamal,

Thanks for your careful review.

The design team tried to stay away from potentially contentions 
discussion by putting firm recommendations in place. Thus currently the 
document is more of "things to think about" and some tradeoffs, and less 
of recommendations (and no requirements).

Alia has asked for more of a list of choices for the protocol designers, 
and this is definitely something which folks can help with as we 
continue this work in the RTGWG. But I also think we need to let the 
specific in-flight WGs (NVO3, SFC, and BIER) use this document as input 
but figure out their own tradeoffs and answers.
>
> The document needs at least one more pass.
Agreed. We're looking for more feedback in the RTGWG and also from the 
WGs that work on the encapsulations.

>
> I have some minor concerns about this document that I believe are 
> resolvable.
> Annotated comments attached.
>     o  SFC carries service meta-data which might be modified or
>        unmodified as the packets follow the service path.  SFC talks of
> Being a little picky, how about:
> "SFC carries service meta-data which might be modified as the packets
> follow the service path."
Agreed - the "unmodified" is a bit to clever (intended to capture that 
in some cases things might be restored to their original).

> 6.  Terminology
>
>     The capitalized keyword MUST is used as defined in
>     http://en.wikipedia.org/wiki/Julmust
>
> Missing the context on what looks like a high calorie delicious drink.
> and should that be https?;->
>

Since you are the first to discover the lack of https in that URL, you 
get a free beverage of your choice (in the bar in Prague if you will be 
there).

>     The UDP source port might change over the lifetime of an encapsulated
>     flow, for instance for DoS mitigation or re-balancing load across
>     ECMP.
> Shouldnt the above statement bear a little more discussion/comment?
> What happens to packet ordering then?
>
I'll add a flag about this.

>        [Note: For any given bit in BIER (that identifies an exit from the
>        BIER domain) there might be multiple immediate next hops.  The
>        BIER entropy field is used to select that next hop as part of BIER
>        processing.  The BIER forwarding process may do equal cost load
>        balancing, but the load balancing procedure MUST choose the same
>        path for any two packets have the same entropy value.]
> "... two packets that have the same ..."
>
Fixed.

>     o  In the case of IP transport use >=14 bits of UDP source port, plus
>        outer IPv6 flowid for entropy.
>
> Looks like a typo.  <=14 bits?
>
No; 14 or 16 depending on the environment. I'll reword as such.

>     o  Reuse Ethernet types - makes it easy to carry existing L2 and L3
>        protocols including IPv6, IPv6, and Ethernet.  Disadvantages are
>        that it is a 16 bit number and we probably need far less than 100
>        values, and the number space is controlled by the IEEE 802 RAC
>        with its own allocation policies.
> If i understood correctly what "reuse" implies: you are suggesting a new
> super-ethertype whose content space will carry an additional type
> semantic so you never have to go back to IEEE?
>
Nope. Rewording as "Use the Ethernet type space" should make it clearer 
- means going to the IEEE RAC for each new protocol. (Not my idea - IDs 
have suggested using the Ethernet type space.)

I'll apply s/Reuse/Use/ to the next bullets as well.

>     Many Internet protocols use fixed values (typically managed by the
>     IANA function) for their next-protocol field.  That facilitates
>     interpretation of packets by middleboxes and e.g., for debugging
>     purposes, but might make the protocol evolution inflexible.  Our
>     collective experience with MPLS shows an alternative where the label
>     can be viewed as an index to a table containing processing
>     instructions and the table content can be managed in different ways.
> Would it not be useful to provide a reference here? Just reading this
> has questions popping for me - who populates this tag-indexed table of
> instructions and could interop be impacted?
>
The DT added the above text based on comments at the mike from Stewart 
Bryant. I don't know if there is any reference for the general concept. 
Anybody else know?

In general this implies some control-plane mechanism to populate the table.

>     o  Would it be useful for the IETF come up with a common scheme for
>        encapsulation protocols?  If not each encapsulation can define its
>        own scheme.
>
> In my view it would be hard to come up with a ring to rule them all.
> There are cases where simple is good enough and asking someone to carry
> a christmas tree is the wrong answer. And, yes, there are cases where
> (to quote Mencken) the answer is clear, simple and wrong (especially
> in one-off-use-cases which then are refactored to fit into square pegs).
> My suggestion is to not be too clever in answering the question above.
>
FWIW the design team just asked the question; someone else gets to 
answer ;-)

But I agree to not blindly forcing some new requirement on common 
next-header approaches across different WGs.

> Reference to Aerolink and the sins committed would be useful.
> I googled aerolink and found references of some radio thing running over IP.
> Given IP provides the fragmention service above, why is aerolink not capable
> of this mechanism? I think there's a simple answer; just reading this didnt help.
There are Aerolink internet-drafts. We'll have to go back and see what 
makes sense as citations.

>     o  Make OAM packets look the same as data packets i.e. the initial
>        part of the OAM payload has the inner Ethernet, IP, TCP/UDP
>        headers as a payload.  (This approach was taken in TRILL out of
>        necessity since there is no UDP header.)  Any OAM bit in the
>        encapsulation header must in any case be excluded from the
>        entropy.
>
> Does it make sense to have inband OAM info? i.e carried alongside the
> data (sure request for a path trace doesnt fit; but inband
> healthinfo may fit); in such a case OAM info could be carried in something
> like a TLV.
>
That was the intent. The formatting issue you caught in the bulleted 
list made this less than clear.

>     In real deployment, the security of the underlying network may be
>     considered for determining the level of security needed in the
>     encapsulation layer.  However for the purposes of this discussion, we
>     assume that network security is out of scope and that the underlying
>     network does not itself provide adequate or as least uniform security
>     mechanisms for encapsulation.
> I found the above paragraph awkward to read.  How about simplifying:
> "This document assumes that the underlying network does not itself
> provide adequate or at least uniform security mechanisms for encapsulation.
> The authors understand that the underlying network security could provide
> useful input into the security needs of the encapsulation layer but ignore
> it to provide a focus on the discussion."
>
Let me check whether that was indeed the intent of the text.

>     o  Interaction with packet level security such as IPsec or DTLS
> So would IPSEC not be considered "underlying network security"?
>
Probably not. If you want to ensure e.g. isolation between different 
tenants then you probably want to avoid know plaintext attacks from one 
tenant against another. That means that the security associations need 
to be different for the different tenants' encapsulated traffic. Hence 
the IPsec policy would need to be aware of how the tenants are described 
in the encapsulation somehow.

Stated differently, can't just apply IPsec pixie dust. Requires some 
real work to specify the details.
> Confidentially is one - but what about integrity of the VNI?
That's captured in the Anti-spoofing bullet.

>        *  The criticality of virtual network isolation depends on whether
>           tenants are trusted or untrusted.  In the most extreme cases,
>           tenants might not only be untrusted but may be considered
>           hostile.
> So would confidentiality then become a requirement to address this?
> It is more readable to make suggestions on each issue on what needs
> to be done.
>
I think it is too early to tell. For NVO3-like deployment there has some 
thoughts, but I don't know if we have the same level of understanding 
across encapsulations. And even for NVO3-like deployments it could be 
different - a public provider where anybody can get a virtual machine, 
vs. isolation between different departments in an enterprise NVO3 network.

>>     o  Our collective IETF experience is that succesful protocols get
>>        deployed outside of the original intended context, hence the
>>        initial assumptions about the threat model might become invalid.
>>        That needs to be considered in the standardization of new
>>        encapsulations.
> So whats the recommendation here? Over-engineer in case something is needed
> later?
>
At least provide the extensibility mechanisms so it can be added without 
too much pain.

> 12.  QoS
>
>     In the Internet architecture we support QoS using the Differentiated
>     Services Code Points (DSCP) in the formerly named Type-of-Service
>     field in the IPv4 header, and in the Traffic-Class field in the IPv6
> Its been at least a decade since the change, do you really need to say
> "formerly named ToS"?
>
I dunno. I was told by the LISPers that the RFC 1812 ICMP errors 
requirement is not yet implemented (getting more than 8 bytes past the 
IP header) and that was 20 years ago.

> Is a ref for NCP needed?
Why not? RFC33 seems to be the most pertinent RFC.

>     Extending a protocol header with new fields can be done in several
>     ways.
>     o  TLVs are a very popular method used in such protocols as IP and
>        TCP.  Depending on the type field size and structure, TLVs can
>        offer a virtually unlimited range of extensions.  A disadvantage
>        of TLVs is that processing them can be verbose, quite complicated,
>        several validations must often be done for each TLV, and there is
> I think if you make such strong comments you need to quantify them.
> A TLV is a formal structure with well defined characteristics. You could
> write efficient code to parse, identify and validate TLVs. How is it
> verbose to process etc?
>
>>        no deterministic ordering for a list of TLVs.  TCP serves as an
> The reason deterministic ordering would matter is if there's dependencies
> between the TLVs. If that is a huge need, then the document needs to provide a
> sample space or explanation why that is important.
>
Fair enough; let me check with the co-authors on how to improve those parts.

> The main difference seems to be in the fact that in a list of header
> extensions, the current extension describes the next; whereas in TLVs
> there is no such relationship; otherwise the T in TLV is an extension
> header. One imposes ordering, the other doesnt really.
In the case of IPv6 header extensions there isn't a strong ordering 
requirement; there is a recommended order in which to transmit them but 
a receiver should be more liberal.
Header extensions seem to be set apart from options/TLV/extensions in 
that they use the same next-header space as is used to indicate a 
"terminal" header such as TCP.

>>     o  Flag-fields are a non-TLV like method of extending a protocol
>>        header.  The basic idea is that the header contains a set of
>>        flags, where each set flags corresponds to optional field that is
>>        present in the header.  GRE is an example of a protocol that
>>        employs this mechanism.  The fields are present in the header in
>>        the order of the flags, and the length of each field is fixed.
>>        Flag-fields are simpler to process compared to TLVs, having fewer
>>        validations and the order of the optional fields is deterministic.
>>        A disadvantage is that range of possible extensions with flag-
>>        fields is smaller than TLVs.
> Qualify with "much smaller" maybe?
>
Defer with other TLV comments.

> "a middlebox could first decapsulate, perform some function then encapsulate;
> which means it will generate a new encapsulation header."

Much better - and I've applied your other editorial fixes that I didn't 
explicitly respond to.
> Is there a reference to work which says quiet periods (which i am implicitly
> reading that in the text above) can be used to change the hash selection?
> I would think that one needs to closely observe packet trends to make
> such a decision. So please provide some ref to some scholarly or
> engineering work.
I don't know of citations to such work; perhaps my co-authors have some.
I recall reading about using markers, but that was a long time ago.

> I think this section is well done.
>
> In regards to TLVs, I understand now a little more where the earlier
> comments come from (IMO: you will need to point to a reference to this
> section from the earlier reference).
>
> Having said that, lets weigh out the pros and cons:
> pro:
> TLVs very flexible - almost give you future proofness in terms of extensibility.
> cons:
> Harder to parallelize in hardware.
>
> I think the pro side should be driving things.
> I would say to the hardware folks - get busy now!
>
> I am still unsure why this is hard to do in h/ware given all the benefits.
> At the expense of getting tomatoes thrown at me:
> sounds like there's an extra parsing step of hardware processing to find each
> individual TLVs "fixed offset" and after that you can parallelize.
This is something that should be discussed in RTGWG.

Thanks,
     Erik

> cheers,
> jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 13 Jul 2015 07:52:09 -0400

Erik,
Sorry for the latency - this got buried because i wasnt on the cc so my
filters
gave it low prio.

> On 6/30/15 2:20 PM, Jamal Hadi Salim wrote:

[..]

> The design team tried to stay away from potentially contentions discussion
> by putting firm recommendations in place. Thus currently the document is
> more of "things to think about" and some tradeoffs, and less of
> recommendations (and no requirements).
>
> and this is definitely something which folks can help with as we continue
> this work in the RTGWG. But I also think we need to let the specific
> in-flight WGs (NVO3, SFC, and BIER) use this document as input but figure
> out their own tradeoffs and answers.
>

Generally the message was as you described.
I probably should have used the term "guidelines" instead of
"recommendations".
I was looking for guidelines and i sometimes didnt grok them.
I liked some of the format in section 18. It provides context ("Here's what
a NIC does for TSO")
and then provides guidelines ("optional protocol fields should not contain
values that must
be updated on a per segment basis .."). I realize that this specific
example is
closer to implementation details - which may change (or get obsolete) over
time but it serves
as a good example of what i was trying to say.
I also realize this may be hard to keep consistent across the document - so
take it as input
of where it may make sense to use such formatting.

 6.  Terminology
> Since you are the first to discover the lack of https in that URL, you get
> a free beverage of your choice (in the bar in Prague if you will be there).
>
>
I will be there. I suspect they dont have Julmust ;->

>>  No; 14 or 16 depending on the environment. I'll reword as such.
Ok. Re-reading the first 2-3 paragraphs in section 7 do present the context
well.

> In general this implies some control-plane mechanism to populate the table.

Maybe an addendum to describe that the control-plane or some human would
administer
this would be useful.

> I recall reading about using markers, but that was a long time ago.

It would help i think to have some reference. I am not sure if this applies:
http://simula.stanford.edu/~alizade/papers/conga-sigcomm14.pdf

cheers,
jamal

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 21 Jul 2015 17:21:50 +0200

On 7/13/15 1:52 PM, Jamal Hadi Salim wrote:
> Erik,
> Sorry for the latency - this got buried because i wasnt on the cc so 
> my filters
> gave it low prio.
>
>
> On Mon, Jul 6, 2015 at 1:17 PM, Erik Nordmark <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     discussion by putting firm recommendations in place. Thus
>     currently the document is more of "things to think about" and some
>     tradeoffs, and less of recommendations (and no requirements).
>
>     Alia has asked for more of a list of choices for the protocol
>     designers, and this is definitely something which folks can help
>     with as we continue this work in the RTGWG. But I also think we
>     need to let the specific in-flight WGs (NVO3, SFC, and BIER) use
>     this document as input but figure out their own tradeoffs and answers.
>
>
> I liked some of the format in section 18. It provides context ("Here's 
> what a NIC does for TSO")
> and then provides guidelines ("optional protocol fields should not 
> contain values that must
> closer to implementation details - which may change (or get obsolete) 
> over time but it serves
> I also realize this may be hard to keep consistent across the document 
> - so take it as input

I just realized the subject line saying draft-rtg-dt-encap-02. We did 
add some bullets at the end of most sections in 
draft-ietf-rtgwg-dt-encap-00.
I don't know if those help, but hopefully they can serve as a prompt for 
getting even more concrete guidelines.

>     The DT added the above text based on comments at the mike from
>     Stewart Bryant. I don't know if there is any reference for the
>     general concept. Anybody else know?
>
> Maybe an addendum to describe that the control-plane or some human 
> would administer
> this would be useful.

Let me ask for volunteers to craft some text tomorrow.

> <http://simula.stanford.edu/%7Ealizade/papers/conga-sigcomm14.pdf>

Let me have a look.

Thanks again,
     Erik