Subject: RE: [Syslog-sec] UDP message size issue proposal

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 13 Oct 2004 11:17:06 +0200

Anton,

I agree to your conclusion.

Although I am the one who always voted for larger sizes, the discussion
has shown that this yields to unnecessary complexity. I can satisfy my
needs with a vendor-extension structured data element, which I think
shows the flexibility of the new drafts.

So I support your move. I would tend to even remove the transport
version header. If there is need to, we could always include that in a
later release (e.g. "v1", which would make it clearly distingshable from
the current frame format). I see little need to do so, though.

Regarding -protocol, I think we should still keep an upper limit in the
spec. Even I don't see any reason why a syslog message should go above
16MB. So for -protocol, I propose the following new text:

####
5.1  Message Length

   The maximum length of any syslog message is 16,777,216 octets.  Any
   receiver receiving a larger message MUST discard the message.  A
   diagnostic message SHOULD be logged in this case.

   A receiver MUST be able to accept messages that are 480 octets (or
   less) in length.  A receiver SHOULD be able to accept messages that
   are 65,535 octets (or less) in length.  It is RECOMMENDED that
   receivers be able to accept messages up to the maximum message length
   of 16,777,216 octets.

   If a receiver receives a message within the maximum length, but with
   a length larger than it handles, the receiver MAY discard or truncate
   it.

   A transport mapping MAY define a maximum length below the one
   specified here.  Each transport mapping MUST support a maximum size
   of 480 octets or more.
####

If there is agreement on this, I can post a new version of -protocol.
That version will also include some changes made thanks to Andrew Ross
comments (sent via private mail). I have finished this version. It is
available for immediate posting, but I would like to wait for agreement
on the size issue (at least if that can be reached quickly).

Rainer

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of 
> Anton Okmianski
> Sent: Tuesday, October 12, 2004 11:25 PM
> To: [email-masked]
> Subject: [Syslog-sec] UDP message size issue proposal
> 
> Hi!
> 
> Sorry for a long delay on this issue - I was on a 2 week 
> vacation.  I have spoken with a number of TCP/UDP/IP experts 
> regarding the sizing issue.  I am ready to propose the 
> following changes:
> 
> 1. Syslog-protocol will state that the max message will be 
> defined by the transport layer.  
> 
> 2. Syslog-transport-udp will support messages up to maximum 
> UDP datagram size of 64K. UDP is a very bad choice for large 
> message transmissions, so it does not make sense for us to 
> stretch it by adding our own fragmentation without other 
> transmission control features such as found in TCP.
> 
> 3. Syslog-transport-udp will rely on IP fragmentation and we 
> will get rid of "proprietary" fragmentation which was 
> designed to handle messages over 64K and deal with various 
> non-compliant network hosts.    
> 
> 4. Syslog-transport-udp will recommend sending messages 
> within the boundaries of prevalent MTU size on a given 
> network to be safe. It will recommend Ethernet's 1500 bytes 
> less headers and will draw reader attention to the minimum 
> MTU size hosts on the network are required to support for 
> IPv6 (576 bytes) and IPv6 (1280 bytes).   
> 
> 5. Path MTU discovery may not work robustly and some TCP/IP 
> stacks may not support UDP packets of full 64K size and 
> truncate them.  We will mention this and bite this bullet.  
> We should not restrict the protocol due to incompliant 
> implementations because it is a moving target and penalizes 
> compliant implementations with extra overhead. The above size 
> recommendation would partially deal with this, but leave the 
> final choice up to the administrator.      
> 
> 6. We will get rid of most syslog transport headers for UDP 
> as they will no longer be needed.  The only thing that will 
> be left is the transport protocol version prefixed to every 
> syslog message.  Should we even bother with that?  
> 
> This is a major change to the syslog-transport-udp.  I'd like 
> to get positive feedback before I proceed with this update.  
> The best part is that if we all agree on the above, the next 
> draft will be 1/3 of the size -- easier read for you. :)
> 
> Thanks,
> Anton. 
>     
> 
> _______________________________________________
> Syslog-sec mailing list
> [email-masked]
> http://www.employees.org/mailman/listinfo/syslog-sec
> 
_______________________________________________
Syslog-sec mailing list
[email-masked]
http://www.employees.org/mailman/listinfo/syslog-sec

---

From: "Anton Okmianski" <[email-masked]>
Date: Wed, 13 Oct 2004 10:37:15 -0400

Rainer:

Glad you liked the latest proposal. I received two other votes for it
as well (from Andrew Ross and John Moehrke) and no disagreements.
So, I will go ahead with the proposal. 

I will remove version field from udp transport as well.  I agree that
in the unlikely scenario that we would need a new version of UDP
transport, we can add the version field at the very front which does
not conflict with syslog-protocol headers.  

One clarification question on suggested text below. 

>    specified here.  Each transport mapping MUST support a maximum
size

You really meant: "Each transport mapping MUST support a *minimum*
size of 480 octets or more."  

Or actually, this should probably be:

"Each transport mapping MUST support transmission of messages of at
least 480 octets".  

Note, we are not placing a requirement on sender or receiver here.
Just saying that transport protocol must be able to handle messages of
this size.  Right?

Anton. 

> That version will also include some changes made thanks to 
> Andrew Ross comments (sent via private mail). I have finished 
> this version. It is available for immediate posting, but I 
> would like to wait for agreement on the size issue (at least 
> if that can be reached quickly).
> 
> Rainer
> 
> > [mailto:[email-masked]] On Behalf Of Anton 
> > Okmianski
> vacation.  I 
> > have spoken with a number of TCP/UDP/IP experts regarding 
> the sizing 
> > issue.  I am ready to propose the following changes:
> > 
> > 2. Syslog-transport-udp will support messages up to maximum UDP 
> > datagram size of 64K. UDP is a very bad choice for large message 
> > transmissions, so it does not make sense for us to stretch it by 
> > adding our own fragmentation without other transmission control 
> > features such as found in TCP.
> > 
> > 3. Syslog-transport-udp will rely on IP fragmentation and 
> we will get 
> > rid of "proprietary" fragmentation which was designed to handle 
> > messages over 64K and deal with various
> > 4. Syslog-transport-udp will recommend sending messages within the

> > boundaries of prevalent MTU size on a given network to be safe. It

> > will recommend Ethernet's 1500 bytes less headers and will 
> draw reader 
> > attention to the minimum MTU size hosts on the network are 
> required to 
> > support for
> stacks may 
> > not support UDP packets of full 64K size and truncate them. 
>  We will 
> > mention this and bite this bullet.
> implementations 
> > because it is a moving target and penalizes compliant 
> implementations 
> > with extra overhead. The above size recommendation would partially

> > deal with this, but leave the
> > 6. We will get rid of most syslog transport headers for UDP as
they 
> > will no longer be needed.  The only thing that will be left is the

> > transport protocol version prefixed to every syslog 
> message.  Should 
> > we even bother with that?
> > 
> > This is a major change to the syslog-transport-udp.  I'd 
> like to get 
> > positive feedback before I proceed with this update.
> > The best part is that if we all agree on the above, the next draft

> > will be 1/3 of the size -- easier read for you. :)
> > 
> > Thanks,
> > Anton. 
> >     
> >

---

From: "David B Harrington" <[email-masked]>
Date: Wed, 13 Oct 2004 10:38:09 -0400

Hi,

The RFC2119 keyword usage needs to be tightened up. I suggest:

>From RFC2119:
"MUST   This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification."
"SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course."

MUST is used to specify compliance requirements. Your text " A
receiver MUST be able to accept messages that are 480 octets (or less)
in length." could be interpreted to mean that I am compliant if I
accept only messages of 4 bytes, since 4 bytes would fit within your
"480 octets (or less)" requirement. "up to and including 480 octets"
is clearer (and consistent with the the same restriction in RFC3417). 

SHOULD and RECOMMENDED are equivalent. So I suggest that 
 " A receiver SHOULD be able to accept messages that
   are 65,535 octets (or less) in length.  It is RECOMMENDED that
   receivers be able to accept messages up to the maximum message
length
   of 16,777,216 octets." 
be reduced to 
 " A receiver SHOULD be able to accept messages up to the maximum
message length of 16,777,216 octets." 

If they cannot support that size, which would be a valid reason to
ignore that requirement, they should understand the implications.
We don't need the intermediate SHOULD 64k size in the standard, we
only need the minimum size and the maximum size.

If the WG wants to make the 64k the standard maximum, yet allow
implementors to support up to 16,777,216 octets, then the text should
specify a SHOULD for 64k, and a MAY for 16,777,216 bytes.

Can a receiver truncate the message, or just the payload of the
message? If the WG ever s=chooses to add something to the end of the
message, such as for security, the text as written would prevent that.

I would also suggest reordering this text to move from minimum to
maximum:

####
5.1  Message Length

   A receiver MUST be able to accept messages up to and including 480
octets in length. 

   A receiver SHOULD be able to accept messages up to and including
65,535 octets in length. 
   If a receiver receives a message with a length larger than it
supports, the receiver MAY 
   discard the message or truncate the payload.

   A receiver MAY accept messages up to the maximum message length
   of 16,777,216 octets.

   A transport mapping MAY define a maximum length below the one
   specified here.  Each transport mapping MUST support at least the
minimum size
   of 480 octets, and not exceed the maximum of 16,777,216 octets.
####

Why are we specifying a maximum length allowed, and why MUST a larger
message be discarded? It is not a matter of interoperability if I
ACCEPT a message larger than 16,777,216 octets. Since I can truncate
any message to my maxiumum supported size, it isn't even a problem if
somebody SENDS me a message larger than that size. So why even mention
such a limit? What if ten years from now, a new transport is defined
that efficiently moves messages largee than this, and somebody wants
to support messages larger than 16,777,216 octets? Why MUST this be
prevented?

If the WG chooses 64k as the STANDARD maximum, why isn't a diagnostic
message required for messages that exceed the standard maximum? Why is
it REQUIRED only for messages above 16,777,216 octets? Does this
impact on-the-wire interoperability? Wouldn't it make sense to log any
message received greater than that supported by the receiver, so the
operator realizes that a message was received but the message may have
been truncated or discarded?

David Harrington
[email-masked]

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Rainer
Gerhards
Sent: Wednesday, October 13, 2004 5:17 AM
To: Anton Okmianski; [email-masked]
Subject: RE: [Syslog-sec] UDP message size issue proposal

Anton,

Although I am the one who always voted for larger sizes, the
discussion has shown that this yields to unnecessary complexity. I can
satisfy my needs with a vendor-extension structured data element,
which I think shows the flexibility of the new drafts.

So I support your move. I would tend to even remove the transport
version header. If there is need to, we could always include that in a
later release (e.g. "v1", which would make it clearly distingshable
from the current frame format). I see little need to do so, though.

Regarding -protocol, I think we should still keep an upper limit in
the spec. Even I don't see any reason why a syslog message should go
above 16MB. So for -protocol, I propose the following new text:

####
5.1  Message Length

   A receiver MUST be able to accept messages that are 480 octets (or
   less) in length.  A receiver SHOULD be able to accept messages that
   are 65,535 octets (or less) in length.  It is RECOMMENDED that
   receivers be able to accept messages up to the maximum message
length
   of 16,777,216 octets.

   If a receiver receives a message within the maximum length, but
with
   a length larger than it handles, the receiver MAY discard or
truncate
   it.

If there is agreement on this, I can post a new version of -protocol.
That version will also include some changes made thanks to Andrew Ross
comments (sent via private mail). I have finished this version. It is
available for immediate posting, but I would like to wait for
agreement on the size issue (at least if that can be reached quickly).

Rainer

> Sorry for a long delay on this issue - I was on a 2 week vacation.
I 
> have spoken with a number of TCP/UDP/IP experts regarding the sizing

> 1. Syslog-protocol will state that the max message will be defined
by 
> the transport layer.
> 3. Syslog-transport-udp will rely on IP fragmentation and we will
get 
> will recommend Ethernet's 1500 bytes less headers and will draw
reader 
> attention to the minimum MTU size hosts on the network are required
to 
> 5. Path MTU discovery may not work robustly and some TCP/IP stacks
may 
> not support UDP packets of full 64K size and truncate them.  We will

implementations 
implementations 
> 6. We will get rid of most syslog transport headers for UDP as they 
> transport protocol version prefixed to every syslog message.  Should

> This is a major change to the syslog-transport-udp.  I'd like to get

---

From: "David B Harrington" <[email-masked]>
Date: Wed, 13 Oct 2004 11:09:37 -0400

Hi,

I am concerned that we are in danger of promoting uses of syslog that
will defeat its current usefulness as a human-readable log of system
activity.

Do network operators really believe the 16,777,216 octet size is a
needed improvement to syslog, or are we designing this size in
explicitly for the vendors of applications who want to use syslog as a
programmatic interface? I believe Syslog should be designed to meet
the needs of operators. Most of the discussion ssems to be from
vendors of syslog receivers or applications. Other protocols such as
FTP might be better used for such a specialty use case. 

Does allowing messages of 16,777,216 octets to be accumulated within
the system log make it harder to use some widely-available minimal
text editors, like Notepad, to view the system log? Will having huge
application-specific messages in the system log make it harder for an
operator to locate useful troubleshooting messages within the system
log? Can the information in such a huge message fit within an 80x25
screen, when an operator is troubleshooting network problems and needs
to use a bare-bones terminal attached to the serial port of a device
to view the system log? 

Syslog was designing to be an operator's tool, and Syslog should be
designed to meet the needs of operators.  Will allowing messages of
this size negatively impact its usefulness in the primary use case?  

What do network **operators** think about the need for messages of
this size? Have we deliberately asked them their opinion on this by,
for example, taking a survey or going to NANOG to ask them?

I don't think we should support such large messages in the system
logging protocol.

David Harrington
[email-masked]

-----Original Message-----

Anton,

####
5.1  Message Length

Rainer

I 

by 
get 
reader 
to 
may 

implementations 
implementations

---

From: "Andrew Ross" <[email-masked]>
Date: Thu, 14 Oct 2004 13:35:40 +1300

David, 

You make a good point on the end use of syslog messages. I'm not a big
fan of large messages, I'd be happy limiting it to 64K even over TCP. It
makes it readable by humans and parsable by standard file parsing
programs.

Some standard text editors will only allow a max line length of 4096
characters.

If we are to assume that a "syslog" message is intended to be displayed
on a console, logged to a file and maybe included in some sort of
paging/e-mail alert system, then the message size really needs to be
kept fairly short.

Rainer's experience is that some devices/apps will send very large
diagnostic messages of around 1MB in size.

I can understand the need in this case to be able to handle large
message sizes. However, I don't plan on accepting messages any larger
than 64K in my implementation of -protocol.

A client would have to make a REALLY good case for me to want to accept
larger messages, especially 16MB in size. FTP, or even HTTP POST would
be the best solution in this case. 

Syslog messages are meant for displaying, logging to disk and alerting.
Not large binary style core dump info or database style records.

I would like to see a note in the -protocol spec that encourages
implementers to only send small messages that adhere to the original
spirit of syslog.

Cheers

Andrew

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of David B
Harrington
Sent: Thursday, 14 October 2004 4:10 a.m.
To: 'Rainer Gerhards'; 'Anton Okmianski'; [email-masked]
Subject: [Syslog-sec] Maximum message size 

Hi,

David Harrington
[email-masked]

-----Original Message-----

Anton,

####
5.1  Message Length

Rainer

I 

by 
get 
reader 
to 
may 

implementations 
implementations