Subject: Re: Reopening jumbo frames in IS-IS

From: [email-masked] (Scott Bradner)
Date: Thu, 23 Jun 2005 08:16:19 -0400 (EDT)

Tony reported:
> Sorry, but I was explicitly told to stop work.

Tony is quite correct

Erik then opined:
> My recollection is that the feedback from IEEE 802 was that we shouldn't 
> touch non-standard things like jumbo-frames, and as a result nothing 
> more was done in this area.
> 
> Perhaps we asked a suboptimal question back then; whether IEEE 802 liked 
> the idea of us doing something for jumboframes and IS-IS. Had we instead 
> asked them whether they were going to specify how jumboframes work with 
> IS-IS, we would probably have received a different answer.

my discussions with the IEEE at that time mentioned IS-IS but that 
was not a factor in the exchange - the IEEE said flatly that 
	1/ jumbo frames for Ethernet were not gonna happen in the IEEE
	2/ they did not want the IETF to make a standard that
	   defined or assumed jumbo frames on Ethernet

re: Erik's suggested question - in effect they did answer that question
'no - they were not going to specify how jumboframes work at all
(nevermind with IS-IS) and please do not that that answer to mean that
the IETF should do so'

Scott

---

From: Eric Rosen <[email-masked]>
Date: Thu, 23 Jun 2005 10:44:17 -0400

> they were not going to specify how jumboframes work at all
> the IETF should do so

Really, with regard to jumbo frames  the marketplace has spoken and the IEEE
has lost. Jumbo frames are a  defacto standard.  (Of course, just as happens
in theIETF,  the IEEE folks don't always  admit it when they've  lost out to
the marketplace.)

Given that  jumbo frames are  in use,  it seems silly  to say that  we can't
specify how to use them for IP protocols.  There is even some precedent; RFC
3032 (MPLS encaps)  mentions the possibility of using  ethernet frames which
contain 1504 or 1508 bytes.  

It's even sillier to say that (a) we're not allowed to do a carefully scoped
standard that solves a well-specified and immediate problem in the use of an
existing de  facto standard, but that (b)  it's fine to do  a standard which
tells folks to ignore IEEE entirely and do bridging in a completely new way.

---

From: Tony Li <[email-masked]>
Date: Thu, 23 Jun 2005 08:48:40 -0700

Scott,

> 	   defined or assumed jumbo frames on Ethernet

My understanding further extended that prohibition to any informational
or experimental RFC as well.  Those would have been (and still would be)
the best possible outcomes.  There was no requirement to make IS-IS over
jumbo frames into a standard.

Tony

---

From: [email-masked] (Scott Bradner)
Date: Thu, 23 Jun 2005 11:53:04 -0400 (EDT)

> or experimental RFC as well.

that was the case to start with, but as someone else pointed out,
a few years later the IEEE said that they would accept an info RFC
(thought they were not all that happy)

Scott

---

From: Tony Li <[email-masked]>
Date: Thu, 23 Jun 2005 09:05:13 -0700

> (thought they were not all that happy)

First I hear of it.  In any case, I would very much like to progress the
work.  Given the non-standard status of jumbo frames, I have no problem
moving forward with informational or experimental as the goal.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 23 Jun 2005 12:49:10 -0400

> tells folks to ignore IEEE entirely and do bridging in a completely new way.

I have to agree that it is silly that IEEE does not acknowledge jumbo
frames.  The provider market has spoken and any large router or large
ethernet switch either implements jumbo frames or doesn't play in that
market.

The reason for jumbo frames is that packet fragmentation in the core
can destroy performance.  As long as MTUs larger than 1500 are in use,
and they are, and the provider POPs where they are handling very high
throughput interfaces of higher MTU, the ethernet MTU will be large
enough or ethernet won't be used.  Its as simple as that.

Lets not use the argument that IEEE should let IETF do things with
jumbo frames because they are allowing IETF to do something else.

Perhaps IETF can specify how ISIS works with the defacto ethernet-like
link layer protocol that happens to differ only in the MTU size and
make it informational.  Its not as if lack of an RFC has stopped any
implementations or deployments so far the benefit of anyone else that
wants to play it wouldn't hurt to provide an informational document.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 23 Jun 2005 12:59:17 -0400

> jumbo frames into a standard.

I think we should have just defied that and written and informational
document.  Its kind of silly for the IETF to expect vendors to keep
implementing toward expired internet-drafts long after a defacto
standard exists as is done all too often.

Lets face it.  Jumbo ethernet frames have been around and have been
widely implemented for about 10 years now.

Curtis

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 24 Jun 2005 22:37:18 +0200

> moving forward with informational or experimental as the goal.

This sounds like information to me, not like an experiment. So I'd support
the idea of aiming at an Informational RFC.

(Peanut gallery question - any other Informationals needed for jumbo frames,
or is IS-IS the only one that needs documenting?)

    Brian

---

From: Tony Li <[email-masked]>
Date: Fri, 24 Jun 2005 16:52:38 -0700

> the idea of aiming at an Informational RFC.

Thanks Brian.

> or is IS-IS the only one that needs documenting?)

I'm not aware of other significant protocols that are not using an
Ethertype encapsulation at this point, so no.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 26 Jun 2005 12:00:44 -0400

In message <[email-masked]>
> or is IS-IS the only one that needs documenting?)

Brian,

It probably would not hurt to have an informational draft describing
jumbo frames, the reason for needing a them, when to use them and
importantly when not to use them.

Jumbo frames exists only to avoid the performance drop that would
otherwise potentially cripple core networks using GbE or 10GbE and
aggregating higher MTU links.  Since jumbo frames could otherwise be
harmful, it might make sense to ask that equipment be capable of
forwarding jumbo frames but originate standard MTU frames (except when
verifying MTU such as in ISIS) and that end user devices such as NIC
cards never originate jumbo frames.  It should also be clearly stated
that jumbo frames be disabled by default and any equipment supporting
them at the very least put harsh warnings in the equipment's user
documentation about the feature being non-standard and has a potential
to cause interoperability problem if just casually enabled.

The fear of the IEEE folks might be that once word gets out of jumbo
frames a lot of people who don't really need them will be asking for
them and equipment in bridged environments won't be able to deal with
them.  In bridged environments fragmenting isn't a possibility and
today most equipment doesn't support jumbo frames, mainly just the
larger routers and switches sold into the large provider market.

(Peanut gallery answer.  Now all you need is an author.)

Curtis

---

From: "Stephen Sprunk" <[email-masked]>
Date: Sun, 26 Jun 2005 12:10:46 -0500

Thus spake "Curtis Villamizar" <[email-masked]>
> otherwise potentially cripple core networks using GbE or 10GbE
> and aggregating higher MTU links.  Since jumbo frames could
> otherwise be harmful, it might make sense to ask that equipment be
> capable of forwarding jumbo frames but originate standard MTU
> frames (except when verifying MTU such as in ISIS) and that end
> user devices such as NIC cards never originate jumbo frames.

That seems excessively restrictive.  There are many environments today using
jumbo frames (8k-10k MTU) on end hosts without problems.

This also begs for a definition of what exactly a "jumbo" frame is; does an
MPLS label (or several) on the front of a 1500-byte Ethernet frame count?
Does a packet on media with a >1500 byte native MTU count?  Should a host on
FDDI or a direct HDLC/PPP link be limited to 1500 byte packets -- even
though its native MTU is 4470 -- because we fear it might need to talk to an
Ethernet-connected host?

> It should also be clearly stated that jumbo frames be disabled by
> default and any equipment supporting them at the very least put
> harsh warnings in the equipment's user documentation about the
> feature being non-standard and has a potential to cause
> interoperability problem if just casually enabled.

I certainly agree that jumbo frames should be disabled by default, and the
warning should explicitly note that severe problems will result if hosts on
the same subnet have different ideas of what the MTU is.  I personally have
never seen a problem with jumbos when every device on a subnet supports them
and is configured with the same MTU, but perhaps someone else can offer some
enlightening anecdotes.

We might consider adding a means for hosts configured with a "jumbo" MTU to
determine if all of its neighbors (and intermediate L2 devices) can indeed
handle such packets.  If such a mechansim existed, we could allow jumbos to
be on by default since hosts could detect when it was necessary to fall back
to a non-jumbo MTU.

S

Stephen Sprunk      "Those people who think they know everything
CCIE #3723         are a great annoyance to those of us who do."
K5SSS                                             --Isaac Asimov

---

From: Daniel Senie <[email-masked]>
Date: Sun, 26 Jun 2005 13:59:37 -0400

>jumbo frames (8k-10k MTU) on end hosts without problems.

Indeed, it would seem to make sense for communications between front 
end and back end servers (e.g. app server to database cases). In 
controlled environments it should help boost database transactions, 
NFS and whatnot. This was certainly one reason FDDI was used in 
places in the past, as large frame sizes were permitted on that medium.

>be on by default since hosts could detect when it was necessary to fall back

Indeed, OSPF has had such a mechanism for years as a result of the 
existence of LAN media types with disparate MTUs (token ring, FDDI). 
Routers are permitted to pack LSAs into packets up to MTU size, and 
if another router on the LAN had a different MTU, significant 
problems resulted (and those problems could lie dormant in a live 
network for a long time before things broke, as it turned out).

It would certainly be worthwhile to have some means for testing. The 
trick is when a transport or routing session fails, some generic 
mechanism is really needed to think to ask the remote end for its 
MTU. That could be an ICMP message asking for MTU. Keep in mind, 
however, that a station which can't receive a frame due to MTU 
mismatch (recipient had MTU 1500, sender something bigger) will not 
even hear the frame (dropped at MAC level) so it can't respond with 
an error message.

The Ethernet MTU of 1500 allowed this issue to be ignored for a long 
time. Other media had issues with it, but they were mostly ignored as 
Ethernet dominated. It'd sure be nice if this were dealt with on a 
generic basis. Probably could have been piggybacked on ARP responses 
if there'd been a range of MTUs to worry about when ARP was proposed. Oh well.

---

From: Tony Li <[email-masked]>
Date: Sun, 26 Jun 2005 12:43:36 -0700

> aggregating higher MTU links.

Another, and perhaps more common, reason for jumbo frames is to support
high speed data center and storage applications.  Many modern systems
have an 8KB primary memory page size, and can be most efficient at bulk
transfers when they can make use of the page as a transfer unit.

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 09:53:41 -0400

Sorry for the late response.  Been away from email.

In message <015a01c57a75$4e1e7b70$6801a8c0@stephen>
> though its native MTU is 4470 -- because we fear it might need to talk to an

Jumbo frames specificly refers to ethernet.  FDDI, HIPPI, HDLC, PPP,
etc are not affected.  It was these interfaces at end systems with
100baseT, GbE, and 10GbE in provider POPs that created the requirement
for ethernet jumbo frames.

No one in their right mind (note that I this is a qualified "no one")
would bridge a PPP link to an ethernet link.  Standards should
discourage doing incredibly stupid things rather than try to
accomodate people who do them with no good reason for doing so.

TCP and most other modern protocols handle the end system MTU quite
nicely and TCP path MTU discovery is widely implemented and easy
enough to enable though I think it is still disabled by default in
most implementations.

> be on by default since hosts could detect when it was necessary to fall back

If used in a SP network, the SP generally knows what is connected much
more so than in an enterprise environment where anyone could connect
anything.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 09:59:31 -0400

> transfers when they can make use of the page as a transfer unit.

Excluding NFS and a few other less used protocols there is little
reason for end system jumbo frames.  It is widely understood that NFS
8K UDP packets are a bad design.

IMHO endorsing jumbo frames in end systems goes to much against what
IEEE would like to see and does so for the benefit of protocols that
should have a new revision that eliminates the need by being more MTU
aware.  This also can improve their performance radically when used
over a WAN or VPN.

Curtis

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 15 Jul 2005 16:55:06 +0200

Curtis Villamizar wrote:
> reason for end system jumbo frames.

There is lots of practical evidence that this isn't true; it's one of
the reasons that Infiniband is very successful in environments such
as high performance redundant database clusters, for example.

Not that this is relevant to the IS-IS issue.

> It is widely understood that NFS

That is irrelevant to the IS-IS issue.

> aware.  This also can improve their performance radically when used

But the typical use of jumbo frames is in highly tuned LAN-based clusters.

Not that this is relevant to the IS-IS issue.

Also, we're not trying to rewrite the IEEE standard; we're trying to deal
with practical reality in the data center.

    Brian

---

From: Radia Perlman <[email-masked]>
Date: Fri, 15 Jul 2005 10:05:55 -0700

>aware.  This also can improve their performance radically when used
What are the technical reasons that IEEE does not like large packets?

Radia

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 13:20:00 -0400

In message <[email-masked]>
> What are the technical reasons that IEEE does not like large packets?

Is existing hardware a technical reason?  It seems to be a good excuse
to not change things in incompatible ways in the IETF so I'd think
that IEEE could use the same reason.  At the very least, disabling by
default should be a requirement (again imho).

I was only aware of the long standing ISP issue with jumbo frames but
not the data center issues that Tony and Brian brought up so I'll step
aside on further conversations about whether jumbo frames are needed
in end systems.

Curtis

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 Jul 2005 10:27:16 -0700

> What are the technical reasons that IEEE does not like large packets?

I can't speak for IEEE, but the reasons usually brought up include  
implementation costs in terms of buffer depths, and mutual jitter  
between competing traffic streams. If you have a session that sends a  
packet every 20 ms and depends on that being mostly maintained,  
having another session send packets that are 30 ms long and can get  
several into the queue ahead of you can be a real pain.

---

From: [email-masked] (Scott Bradner)
Date: Fri, 15 Jul 2005 13:30:19 -0400 (EDT)

also - there is no layer-2 way to say "packet too big"
so that big packets sent through paths that do not support big packets
go into a black hole and the software never knows what happened

Scott

----
>From [email-masked]  Fri Jul 15 13:27:20 2005
Cc: [email-masked], [email-masked], [email-masked],
	[email-masked], Brian E Carpenter <[email-masked]>,
	Scott Bradner <[email-masked]>
From: Fred Baker <[email-masked]>
Subject: Re: Reopening jumbo frames in IS-IS
Date: Fri, 15 Jul 2005 10:27:16 -0700
To: Radia Perlman <[email-masked]>
IIM-SIG: v:"1.1"; h:"imail.cisco.com"; d:"cisco.com"; z:"home"; m:"krs";
	t:"1121448351.73505"; x:"432200"; a:"rsa-sha1"; b:"nofws:421";
	e:"Iw=="; n:"sQYarK2E51MdcTiUqeif3F7cWdxIfoCiXhdfb9vD5ee/j0jXL15gbFxF2p"
	"XIweAblu0N6XAgK7k+wrbr7bQDJaCDqOmzqpRUBjIRQAXQ7NzadpmR3pUL6wxaRUtW+c43sl9jC"
	"50Qg1sXHpPjt8Y+Y16ioyQAQAdSunM4YhevURc=";
	s:"PRq9DVgsHUgdLZP6sRxe94EDYyx7rXYtgeFGzLriIX75ui9ef1uLnh7XQ1nYkEgZgUbgUrce"
	"j9ixfAHCKzd2oKC2SPQxG2Vnw1cm9o0mM/fNAmN9DIstyE5Px0hi+06+oVrzgFpZxxJnerl1okq"
	"hNk5G8/GFnmG/344JKdaVCXE=";
	c:"From: Fred Baker <[email-masked]>";
	c:"Subject: Re: Reopening jumbo frames in IS-IS";
	c:"Date: Fri, 15 Jul 2005 10:27:16 -0700"
IIM-VERIFY: s:"y"; v:"y"; r:"60"; h:"imail.cisco.com";
	c:"message from imail.cisco.com verified; "

---

From: Bob Hinden <[email-masked]>
Date: Fri, 15 Jul 2005 11:01:01 -0700

Radia,

>What are the technical reasons that IEEE does not like large packets?

I can't speak for IEEE, but I have always thought that one of there reasons 
that Ethernet has been so successful is that that IEEE tried very hard to 
insure backward compatibility between the different versions.  It made it 
easier to bridge between 10M/100M/1G/10G/etc. versions, new versions didn't 
break any protocols that ran over Ethernet, and it is easier to build NICs 
that supported a range of variants.

It also avoided having to build things like a FDDI/Ethernet bridge I once 
heard about that supported IP fragmentation.  I bet you remember that :-)

Bob

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 12:59:28 -0500

> default should be a requirement (again imho).

The IETF also has a history of allowing incompatible changes in protocols 
_provided that there is a mechanism to first determine if those changes are 
supported on both ends_.

It seems fair to me to allow jumbos to be on by default iff there were a 
standard means to determine that the other host also supports them.

> aside on further conversations about whether jumbo frames are needed

I am mainly familiar with jumbos used in HPC, though I've also seen them 
used on "back side" networks used for backups or file serving. 
Communication between application and DB servers is similar and doesn't 
surprise me.  However, all cases I'm aware of are closed networks with 
processes for _humans_ to ensure that all hosts support a nonstandard MTU. 
Jumbos can't move to general use until humans are removed from the loop.

Also, there are large numbers of routers (and "L3 switches") that are 
limited in the number of pps they can handle.  Jumbos allow a higher 
bandwidth for the same pps.  OTOH, as more applications with small packets 
(e.g. VoIP) come around, we may need jumbos just to maintain current pps 
levels on a given pipe.

S

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 12:59:35 -0500

>> That seems excessively restrictive.  There are many environments
>> today using jumbo frames (8k-10k MTU) on end hosts without
>> problems.
>>
>> This also begs for a definition of what exactly a "jumbo" frame is;
>> does an MPLS label (or several) on the front of a 1500-byte Ethernet
>> frame count?  Does a packet on media with a >1500 byte native
>> MTU count?  Should a host on FDDI or a direct HDLC/PPP link be
>> limited to 1500 byte packets -- even though its native MTU is 4470
>> -- because we fear it might need to talk to an Ethernet-connected
>> host?
>
> 100baseT, GbE, and 10GbE in provider POPs that created the
> requirement for ethernet jumbo frames.

Only as it relates to IS-IS; jumbos existed long before in other contexts 
despite the IEEE's efforts to the contrary.

> accomodate people who do them with no good reason for doing so.

Then a large number of ISPs are out of their minds...  The two main ways to 
deploy ADSL service are (a) Ethernet bridged to PPPoATM, and (b) PPPoE 
directly to the host.  I think the latter is also commonly used for DOCSIS.

If bridging POS and Ethernet were safer (i.e. jumbos could be assumed), we'd 
probably see that become common in ISPs as well.

> enough to enable though I think it is still disabled by default in

Win2k and later have PMTUD on by default, but unfortunately not PMTU black 
hole detection.  I'm not sure about earlier Windows versions or 
MacOS/Linux/etc.

However, there's no current means for MTU detection for hosts on the same 
subnet.  I can envision a number of ways to fix this for IP traffic, but I 
don't know enough about IS-IS to know if any of them are applicable.

S

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 Jul 2005 12:59:26 -0700

On Jul 15, 2005, at 10:59 AM, Stephen Sprunk wrote:

> It seems fair to me to allow jumbos to be on by default iff there  
> were a standard means to determine that the other host also  
> supports them.

The issue isn't the other host; there usually is a way to determine  
that, like a TCP MSS negotiation. The issue is the network - is there  
some buried switch that won't support them? Is there a tunnel in the  
path that reduces the MTU below what the end hosts expect?

I tend to like draft-ietf-pmtud-method-04.txt's proposal, although I  
tend to think it is a bit over-optimized. Basically, send a large  
packet every so often and see if it gets through. If it demonstrably  
does, you can decide to use them in regular conversation. Otherwise,  
do what works. Thinking in terms of TCP, we now suggest that TCP  
sessions send 2-4 segments in their first burst. well, gee. What if  
the session sent a 9000 byte packet plus three 1460 byte packets? The  
Ack would either report something 9K into the session or something 4K  
into the session. Seems like an excellent start to me... Or, have the  
delivery API include with it the size of the largest fragment folded  
into a delivered datagram. If the number falls, the receiving TCP  
could renegotiate the MSS. Maybe that's too simple.

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 15:01:21 -0500

Thus spake "Bob Hinden" <[email-masked]>
> I can't speak for IEEE, but I have always thought that one of there
> reasons that Ethernet has been so successful is that that IEEE tried
> very hard to insure backward compatibility between the different
> versions.  It made it easier to bridge between 10M/100M/1G/10G/etc.
> versions, new versions didn't break any protocols that ran over Ethernet,
> and it is easier to build NICs that supported a range of variants.
>

Wait a minute...  If Ethernet supported jumbo frames, the FDDI-Ethernet 
bridge wouldn't have needed to support fragmentation -- just set the 
Ethernet side's MTU to 4470 and all would be well.

I dealt with this many times when bridging Token Ring and Ethernet, and the 
only solution that worked in every case was to drop the Token Ring's MTU 
(network-wide, since TR implies SRB) down to 1500 -- a horrible kludge. 
Sometimes other tactics worked, including dropping oversized frames with no 
fragmentation, but some SNA apps were really touchy about that.  IP handled 
things a bit better, but still not as well as jumbos would have.

S

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 17:01:38 -0400

In message <023c01c5896a$0ede1020$6501a8c0@dax>
> directly to the host.  I think the latter is also commonly used for DOCSIS.

Agreed.  :-)

These were more the choices of DSLAM vendors and to a less extent the
RBOCs/ILECs who were latecomers.  The traditional ISPs at the time
preferred that the DSLAMs just do IP over PPP on the customer router
or host and IP over PPP or ATM on the backhaul side.  The DSLAM
vendors thought that by matching the ATM parameters to the DSL rate
they could all but eliminate buffering in the DSLAM.

> probably see that become common in ISPs as well.

Has it become that bad already? :-)

Who was it that said the Internet is growing exponentially but the
amount of clue is growing linearly at best so the clue density is
shrinking exponentially?  I think it was Randy Bush in the early 90s.

> CCIE #3723         are a great annoyance to those of us who do."

ISIS sends hellos at full MTU.  If the other end can't handle them,
then no neighbor adjacency comes up.  There is no L2 DF bit and ICMP
would fragment reply as there is in IP so TCP doesn't handle the MTU
mismatch on the same subnet.  Something like that (an L2 DF and ICMP)
would be needed to handle two devices with an MTU mismatch on the same
subnet or some probe packet of the type ISIS uses.  The diagnostic
tool available to today's operator is "ping".  Use the -s option.

Until something changes there is no automatic means to detect and
correct an MTU mismatch on the same subnet.  If running ISIS there is
the means to detect the error but I don't think there is an automatic
correction (no MTU fallback).  I'd have to check the expired draft.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 16:25:36 -0400

In message <[email-masked]>
Fred Baker writes:

> several into the queue ahead of you can be a real pain. 

btw- 30 msec at 1 Gb/s is 30 mbit or about 4mB so the delay jitter
argument falls apart for GbE.  Its 400 kB for 100baseT.  It is a
problem for 10baseT and 40 kB in the queue.  For chips that put on the
order of 10 packets into an on chip hardware queue and don't consider
the length of the packets in the queue, 4-8KB MTU might get you 30
msec with 10baseT and 5-10 packets in the hardware queue.  The IEEE
ethernet types will remind us that a GbE and 10baseT might be bridged
together.  Still the jitter buffer argument is a very weak argument.

The memory requirements is not really an issue either.  You need the
same buffer depth (a lot more than a few packets) for any MTU.

Therefore the cost of changing the implementation is the only cost.
The cost of interoperability problems is an operational cost that
needs to be considered and is the reason to procede with caution.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 17:15:59 -0400

In message <026601c58978$13564970$6501a8c0@dax>
> CCIE #3723         are a great annoyance to those of us who do."

Maybe the problem is that bridging dissimilar interfaces is itself a
horrible hack.  That's what routing is for.  Yes, some protocols don't
support routing but they are now dinosaurs, SNA included.  I thought
the days of building networks around the needs of netbios, netware 3,
and sna were over.

Curtis

---

From: Tony Li <[email-masked]>
Date: Fri, 15 Jul 2005 14:39:01 -0700

> correction (no MTU fallback).  I'd have to check the expired draft.

There is currently no such mechanism.

Note that correction between routers is less of an issue.  If all end
systems on the (V)LAN are not agreed on the MTU, then there is a more
general problem.  If all end systems are fixed, then presumably all
routers are as well.

Tony

---

From: Bob Hinden <[email-masked]>
Date: Fri, 15 Jul 2005 14:54:11 -0700

Stephen,

>Ethernet side's MTU to 4470 and all would be well.

That was back in the days of 10M Ethernet and it didn't support jumbo 
frames.  Seems to me the issue would still occur unless both FDDI and 
Ethernet (w/ jumbo frames) both had the same max MTU.  Someone would want 
to use the max of the bigger of the two.

Bob

p.s. I also find it ironic that in the other side of the house there are 
folks who think 53 byte packets are ideal :-)

---

From: Tony Li <[email-masked]>
Date: Fri, 15 Jul 2005 15:06:10 -0700

>> Wait a minute...  If Ethernet supported jumbo frames, the
>> FDDI-Ethernet bridge wouldn't have needed to support fragmentation --
>> just set the Ethernet side's MTU to 4470 and all would be well.
> 
> Ethernet (w/ jumbo frames) both had the same max MTU.  Someone would
> want to use the max of the bigger of the two.

While I appreciate the free trip down memory lane, I'm kind of missing
the point of the discussion here.  I think that we all now acknowledge
that Jumbo Frames exist, that like it or not, they are in use.  The
question now is what are we going to do about it...

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 09:53:41 -0400

Curtis

_______________________________________________
routing-discussion mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/routing-discussion

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 Jul 2005 10:27:16 -0700

---

From: Bob Hinden <[email-masked]>
Date: Fri, 15 Jul 2005 14:54:11 -0700

Stephen,

Bob

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 12:59:35 -0500

S

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 16:25:36 -0400

Curtis

---

From: Tony Li <[email-masked]>
Date: Fri, 15 Jul 2005 15:06:10 -0700

Tony

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 09:59:31 -0400

Curtis

---

From: Tony Li <[email-masked]>
Date: Sun, 24 Jul 2005 07:55:48 -0700

Curtis,

> aware.  This also can improve their performance radically when used

I would agree that using UDP for NFS in the first place was a
questionable call, exacerbated when UDP checksums were not enabled by
default on the implementations in question.

However, the real issue here is the end system efficiency, not MTU
awareness.  With a larger LAN MTU, the end system can be more efficient
by transfering an entire page at a time, rather than fragmenting it at
the UDP level or at the TCP level.  For some systems, this is a
non-trivial consideration.  When you couple this with ever increasing
link speeds (e.g., 10GigE), there seems little reason to preclude using
larger MTU sizes.  You'll note that the supercomputer folks have been
pushing us in this direction for years and we also had this discussion
relative to IPv6.

As to what the IEEE would like to see, that would seem to be a lesser
consideration than delivering the technology that our customers (i.e.,
end users) would like to see.  Those votes have already been cast and
are clearly in favor of jumbo frames.

The IETF's larger job, besides pushing this work forward, should be to
consider the role and scope of the MTU in the Internet in general.  It
is a fine point of the Internet architecture, and we have been remiss in
  not attending to this issue.  Right now, much of the Internet supports
a 4096 byte MTU, thanks primarily to some predominant POS
implementations.  However, most access links are still constrained to
1500B.  It would be very helpful, operationally, if the IETF could come
up with a recommended set of MTUs and set some expectations for end
users on the predominant MTU sizes.

While many folks may feel that this is a trivial and pointless detail,
it has a decided economic impact.  A fine example is classical IP over
ATM.  There the IETF specified a large default MTU (9120B, if memory
serves).  This caused equipment prices to rise as packet buffering
memory needed to be sized accordingly.  However, due to the lack of
transit capability for this MTU size, this capability has been largely
wasted.  Admittedly, this is indeed trivial with respect to the overall
waste on ATM technology, but it would be best if we did not repeat this
particular experiment.

Tony

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 15 Jul 2005 15:01:21 -0500

S

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 15 Jul 2005 17:01:38 -0400

Agreed.  :-)

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 24 Jul 2005 22:08:01 -0400

> pushing us in this direction for years and we also had this discussion

That was then and now is now.  Network interfaces are now faster than
disk drives by an order of magnitude or more.  Yes, there is probably
some Cray out there front ended with an Amdol putting 4K frames ont a
HIPPI interface because at the time breaking up the disk block by
anything but a fixed small factor of two was unthinkable.

> are clearly in favor of jumbo frames.

At least we agree on the conclusion.

> users on the predominant MTU sizes.

Unfortunately in the past each new interface seems to want a bigger
MTU than the last.  I'm not sure where the 9K+ for IP/ATM came from.
I don't think its at all likely that the IETF will be going back and
changing MTUs just to make them all the same.  Why would they?  So
that anyone building a new token ring or FDDI interface can use the
new MTU?  No one builds token ring interfaces anymore.

> waste on ATM technology, but it would be best if we did not repeat this

That doesn't make sense.  In switches if you tossed out the ATM
traffic management and actually tried to make use of most of the
bandwidth you'd need delay*bandwidth sized buffers and the early
switches had 128-256 cells.  That had nothing to do with MTU.  In
routers the were two big problems.  One was packet buffers which is
unrelated to MTU.  The other was SAR buffering.  The problem existed
when carrying Internet traffic over ATM and then reassembling at the
other end and so also had nothing to do with IP/ATM MTU because there
was no IP/ATM traffic involved.  In NICs if cells didn't get reordered
(which shouldn't happen) then you'd need at most a packet per
concurrent open connection.  If you had thousands of concurrent open
connections (such as in a web server) you'd need megabytes of SAR
memory.  With an ether NIC and a jumbo MTU you don't need that sort of
memory oncard because you get an entire frame at a time, not small
pieces of many frames.  With jumbo frame ether you still need the TCP
buffers (which should always larger by at least a factor of 4) but
that's in processor main memory.  So it was really hopelessly
inadequate buffering in switches and problems with SAR on routers and
NICs and not the large MTU that was ATM's problem.

> Tony

We're straying off topic.  We agree on the conclusion that IETF should
support jumbo frames for ethernet.  If you bringing up further
examples to support extending the jumbo MTU to end systems and not
just routers and switches, that's fine with me.

Curtis

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 24 Jul 2005 20:32:52 -0700

On Sun, Jul 24, 2005 at 10:08:01PM -0400, Curtis Villamizar wrote:
> disk drives by an order of magnitude or more.

But that's not necessary the right metric to judge the cost of
packetizing. Distributed applications can easily shuffle data just between
memory without having to resolve to disk transfer. I am not sure if
there are any research community agreed upon metrics (sounds like a
question to Scott). I personally would look for packetization to be
acceptable as long as it takes up less % of CPU than the DMA of the data
takes up in % of memory bandwidth. 

Just observing from past experience - not really wanting to make a
statement one way or the other about research computers nowadays.

Just for completeness (not that i'd expect anyone to care):
For multicast to be able to leverage higher MTU than 1280 effectively and
safely would require something very close to the architectural minimum limit
to be raised (aka: ubiquitous availability of larger MTU across the Internet,
end-to-end).

So, i would certainly be happy if the agreed upon minimum MTU even in
ethernet was raised to something like let's say at minimum the 4k so
common in core networks anyhow (which also happens to be larger than many
video frame sizes, so that could avoid fragmentation on many typical video
encoders).

Cheers
	Toerless

---

From: Tony Li <[email-masked]>
Date: Mon, 25 Jul 2005 06:25:59 +0100

> just routers and switches, that's fine with me.

Shutting the hell up...

Tony