Subject: [Syslog-sec] RE: Maximum message size

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 14 Oct 2004 13:34:24 +0200

David and all,

many thanks for all the good comments.

I agree that 16 MB for a single message is far more than actually should
be used. Let's look a bit at the history of this:

Initially, we noticed that the 1024 octet limit for current syslog is
too short. I think there is still general agreement on this. We had a
good discussion about the size issue in September 2003. A link to an
archive is here:

http://www.syslog.cc/ietf/autoarc/msg00855.html

The general issue is that 1K is too short for many applications. For
example, Marshall Glen mentions the healthcare needs (RFC3881) for a
larger message. As far as I know, the IHE initiative currently
standardizes on a maximum XML stream size of 32K (which is then supposed
to be transmitted via syslog). I think a message size limit of 64K would
probably be suffcient for a long time to come (and leaves some headroom
for message content increases).

Besides that, I myself identified a need - in very rare occasions - to
send information larger than 64K. In extreme cases, this information
could be as large as one megabyte. Again, these were cases that might
happen once in a year. All of this, of course, only if an operator
*really* wants to submit that large information. So the initial approach
and suggestion was to introduce a way to build "message groups". That
would have been a way to split "oversize messages" (as they were called
then) into multiple syslog messages. They key to that concept was that
each single message could be kept within a reasonable limit (it would
have worked even with 1K max message size). It would only have been a
way to transmit very large messages in those very seldom cases that they
would actually be used.

Some time later, we came to the conclusion that it should be possible
for a single syslog message to be of the largest imaginable size. It was
argued that fragmentation is a transport issue and as such
application-level "fragmentation" (the message groups) would not do
good. We agreed to that.

Even some time later, we needed to define a maximum message size, just
to limit it somewhere (if I recall correctly, that was a transport
requirement). We picked 16MB 
just as a number that we thought would never have be reached and such be
sufficient for all imaginable time to come. As far as I was concerend,
that decision was driven by the fact that in the past there were often
limits that nobody expected to reach - and yet they wer quickly reached.
Som 16MB was picked more or less as "larger than anybody will need
anytime in the forseeable future". We did not even have a big discussion
on that limit.

I think this are the key points on how we reached this limit. Now,
thanks to your call and looking the whole situation, I really think
we've gone overboard. That 16MB limit is something nobody intends to
use. Even a lower - but high limit - is extremely unlikely. I'd expect
to see a 1MB "message" in maybe 0.0001% of the cases - if at all (but I
expect to see e.g. 70KB "message" in maybe 0.001% of the cases).

Also, I have read your proposal to set a maxium size for the message,
but allow an implementation to go beyond that. Maybe something like
this:

####
5.1  Message Length

   A receiver MUST be able to accept messages up to and including 480
octets in length. 

   A receiver SHOULD be able to accept messages up to and including
65,535 octets in length. 
   If a receiver receives a message with a length larger than it
supports, the receiver MAY 
   discard the message or truncate the payload.

   A receiver MAY accept messages larger than 65,535 octets.
####

I've dropped the wording that transports MAY support only a smaller max
size - I think UDP was the most critical in this regard and it obviously
handles 64K. So I assume this clause is not really needed if we go for
64K as the max size.

With that clause, we would limit the size for all interop cases, but
leave a window open so that the message length could be extended if
there is a real need AND the operator and software vendor wants this. I
assume nobody will implement this burden if there is no market need to
do so. But if it is, it clould at least be implemented in a
standards-compliant way.

If I look back at the original requirement - provide a way to transmit
very rare oversize messages, I think we can safely implement this the
bounds of the current proposals, even if the size limit does not go
beyond 64K. If somebody needs it, this can be easily done via an
experimental (or vendor-specific) SD-ID. Again, I do not think somebody
will accept this burden without market need.

In the light of this, I, too, would recommend that we limit the message
size to 64K. I would prefer, however, that we keep the maximum open as
in the suggested wording above.

Rainer

> -----Original Message-----
> From: David B Harrington [mailto:[email-masked]] 
> Sent: Wednesday, October 13, 2004 5:10 PM
> To: Rainer Gerhards; 'Anton Okmianski'; [email-masked]
> Subject: Maximum message size 
> 
> Hi,
> 
> I am concerned that we are in danger of promoting uses of syslog that
> will defeat its current usefulness as a human-readable log of system
> activity.
> 
> Do network operators really believe the 16,777,216 octet size is a
> needed improvement to syslog, or are we designing this size in
> explicitly for the vendors of applications who want to use syslog as a
> programmatic interface? I believe Syslog should be designed to meet
> the needs of operators. Most of the discussion ssems to be from
> vendors of syslog receivers or applications. Other protocols such as
> FTP might be better used for such a specialty use case. 
> 
> Does allowing messages of 16,777,216 octets to be accumulated within
> the system log make it harder to use some widely-available minimal
> text editors, like Notepad, to view the system log? Will having huge
> application-specific messages in the system log make it harder for an
> operator to locate useful troubleshooting messages within the system
> log? Can the information in such a huge message fit within an 80x25
> screen, when an operator is troubleshooting network problems and needs
> to use a bare-bones terminal attached to the serial port of a device
> to view the system log? 
> 
> Syslog was designing to be an operator's tool, and Syslog should be
> designed to meet the needs of operators.  Will allowing messages of
> this size negatively impact its usefulness in the primary use case?  
> 
> What do network **operators** think about the need for messages of
> this size? Have we deliberately asked them their opinion on this by,
> for example, taking a survey or going to NANOG to ask them?
> 
> I don't think we should support such large messages in the system
> logging protocol.
> 
> 
> David Harrington
> [email-masked]
> 
> 
> 
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Rainer
> Gerhards
> Sent: Wednesday, October 13, 2004 5:17 AM
> To: Anton Okmianski; [email-masked]
> Subject: RE: [Syslog-sec] UDP message size issue proposal
> 
> Anton,
> 
> I agree to your conclusion.
> 
> Although I am the one who always voted for larger sizes, the
> discussion has shown that this yields to unnecessary complexity. I can
> satisfy my needs with a vendor-extension structured data element,
> which I think shows the flexibility of the new drafts.
> 
> So I support your move. I would tend to even remove the transport
> version header. If there is need to, we could always include that in a
> later release (e.g. "v1", which would make it clearly distingshable
> from the current frame format). I see little need to do so, though.
> 
> Regarding -protocol, I think we should still keep an upper limit in
> the spec. Even I don't see any reason why a syslog message should go
> above 16MB. So for -protocol, I propose the following new text:
> 
> ####
> 5.1  Message Length
> 
>    The maximum length of any syslog message is 16,777,216 octets.  Any
>    receiver receiving a larger message MUST discard the message.  A
>    diagnostic message SHOULD be logged in this case.
> 
>    A receiver MUST be able to accept messages that are 480 octets (or
>    less) in length.  A receiver SHOULD be able to accept messages that
>    are 65,535 octets (or less) in length.  It is RECOMMENDED that
>    receivers be able to accept messages up to the maximum message
> length
>    of 16,777,216 octets.
> 
>    If a receiver receives a message within the maximum length, but
> with
>    a length larger than it handles, the receiver MAY discard or
> truncate
>    it.
> 
>    A transport mapping MAY define a maximum length below the one
>    specified here.  Each transport mapping MUST support a maximum size
>    of 480 octets or more.
> ####
> 
> If there is agreement on this, I can post a new version of -protocol.
> That version will also include some changes made thanks to Andrew Ross
> comments (sent via private mail). I have finished this version. It is
> available for immediate posting, but I would like to wait for
> agreement on the size issue (at least if that can be reached quickly).
> 
> Rainer
> 
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]] On Behalf Of Anton 
> > Okmianski
> > Sent: Tuesday, October 12, 2004 11:25 PM
> > To: [email-masked]
> > Subject: [Syslog-sec] UDP message size issue proposal
> > 
> > Hi!
> > 
> > Sorry for a long delay on this issue - I was on a 2 week vacation.
> I 
> > have spoken with a number of TCP/UDP/IP experts regarding the sizing
> 
> > issue.  I am ready to propose the following changes:
> > 
> > 1. Syslog-protocol will state that the max message will be defined
> by 
> > the transport layer.
> > 
> > 2. Syslog-transport-udp will support messages up to maximum UDP 
> > datagram size of 64K. UDP is a very bad choice for large message 
> > transmissions, so it does not make sense for us to stretch it by 
> > adding our own fragmentation without other transmission control 
> > features such as found in TCP.
> > 
> > 3. Syslog-transport-udp will rely on IP fragmentation and we will
> get 
> > rid of "proprietary" fragmentation which was designed to handle 
> > messages over 64K and deal with various
> > non-compliant network hosts.    
> > 
> > 4. Syslog-transport-udp will recommend sending messages within the 
> > boundaries of prevalent MTU size on a given network to be safe. It 
> > will recommend Ethernet's 1500 bytes less headers and will draw
> reader 
> > attention to the minimum MTU size hosts on the network are required
> to 
> > support for
> > IPv6 (576 bytes) and IPv6 (1280 bytes).   
> > 
> > 5. Path MTU discovery may not work robustly and some TCP/IP stacks
> may 
> > not support UDP packets of full 64K size and truncate them.  We will
> 
> > mention this and bite this bullet.
> > We should not restrict the protocol due to incompliant
> implementations 
> > because it is a moving target and penalizes compliant
> implementations 
> > with extra overhead. The above size recommendation would partially 
> > deal with this, but leave the
> > final choice up to the administrator.      
> > 
> > 6. We will get rid of most syslog transport headers for UDP as they 
> > will no longer be needed.  The only thing that will be left is the 
> > transport protocol version prefixed to every syslog message.  Should
> 
> > we even bother with that?
> > 
> > This is a major change to the syslog-transport-udp.  I'd like to get
> 
> > positive feedback before I proceed with this update.
> > The best part is that if we all agree on the above, the next draft 
> > will be 1/3 of the size -- easier read for you. :)
> > 
> > Thanks,
> > Anton. 
> >     
> > 
> > _______________________________________________
> > Syslog-sec mailing list
> > [email-masked]
> > http://www.employees.org/mailman/listinfo/syslog-sec
> > 
> _______________________________________________
> Syslog-sec mailing list
> [email-masked]
> http://www.employees.org/mailman/listinfo/syslog-sec
> 
> 
> 
_______________________________________________
Syslog-sec mailing list
[email-masked]
http://www.employees.org/mailman/listinfo/syslog-sec

---

From: "David B Harrington" <[email-masked]>
Date: Thu, 14 Oct 2004 08:53:01 -0400

Hi Rainer,

Thanks for making the changes.

I'd like to make one point a little more strenuously:
"For interoperability reasons, all receiver implementations SHOULD be
able to accept messages up to and including 65,535 octets in length."

Also, I think the last two sentences could be combined into one - "If
a receiver receives a message with a length larger than 65,535 octets,
or larger than it supports, the receiver MAY discard the message or
truncate the payload."

dbh

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Rainer
Gerhards
Sent: Thursday, October 14, 2004 7:34 AM
To: [email-masked]
Subject: [Syslog-sec] RE: Maximum message size 

David and all,

I agree that 16 MB for a single message is far more than actually
should be used. Let's look a bit at the history of this:

The general issue is that 1K is too short for many applications. For
example, Marshall Glen mentions the healthcare needs (RFC3881) for a
larger message. As far as I know, the IHE initiative currently
standardizes on a maximum XML stream size of 32K (which is then
supposed to be transmitted via syslog). I think a message size limit
of 64K would probably be suffcient for a long time to come (and leaves
some headroom for message content increases).

Besides that, I myself identified a need - in very rare occasions - to
send information larger than 64K. In extreme cases, this information
could be as large as one megabyte. Again, these were cases that might
happen once in a year. All of this, of course, only if an operator
*really* wants to submit that large information. So the initial
approach and suggestion was to introduce a way to build "message
groups". That would have been a way to split "oversize messages" (as
they were called
then) into multiple syslog messages. They key to that concept was that
each single message could be kept within a reasonable limit (it would
have worked even with 1K max message size). It would only have been a
way to transmit very large messages in those very seldom cases that
they would actually be used.

Some time later, we came to the conclusion that it should be possible
for a single syslog message to be of the largest imaginable size. It
was argued that fragmentation is a transport issue and as such
application-level "fragmentation" (the message groups) would not do
good. We agreed to that.

Even some time later, we needed to define a maximum message size, just
to limit it somewhere (if I recall correctly, that was a transport
requirement). We picked 16MB just as a number that we thought would
never have be reached and such be sufficient for all imaginable time
to come. As far as I was concerend, that decision was driven by the
fact that in the past there were often limits that nobody expected to
reach - and yet they wer quickly reached.
Som 16MB was picked more or less as "larger than anybody will need
anytime in the forseeable future". We did not even have a big
discussion on that limit.

I think this are the key points on how we reached this limit. Now,
thanks to your call and looking the whole situation, I really think
we've gone overboard. That 16MB limit is something nobody intends to
use. Even a lower - but high limit - is extremely unlikely. I'd expect
to see a 1MB "message" in maybe 0.0001% of the cases - if at all (but
I expect to see e.g. 70KB "message" in maybe 0.001% of the cases).

####
5.1  Message Length

I've dropped the wording that transports MAY support only a smaller
max size - I think UDP was the most critical in this regard and it
obviously handles 64K. So I assume this clause is not really needed if
we go for 64K as the max size.

With that clause, we would limit the size for all interop cases, but
leave a window open so that the message length could be extended if
there is a real need AND the operator and software vendor wants this.
I assume nobody will implement this burden if there is no market need
to do so. But if it is, it clould at least be implemented in a
standards-compliant way.

If I look back at the original requirement - provide a way to transmit
very rare oversize messages, I think we can safely implement this the
bounds of the current proposals, even if the size limit does not go
beyond 64K. If somebody needs it, this can be easily done via an
experimental (or vendor-specific) SD-ID. Again, I do not think
somebody will accept this burden without market need.

In the light of this, I, too, would recommend that we limit the
message size to 64K. I would prefer, however, that we keep the maximum
open as in the suggested wording above.

Rainer

---

From: "David B Harrington" <[email-masked]>
Date: Thu, 14 Oct 2004 09:21:24 -0400

Hi,

I have concerns about the requirements for maximum message size, and
would like to propose an alternate approach.

Most syslog messages currently are less than 1K in size. There is
consensus that 1K is typically acceptable, but it is a bit restrictive
in some circunstances and the size should be increased. I agree with
that consensus. I disagree that the size needs to be increased 64
times to accommodate normal usage, and I recognize that some valid,
but abnormal, usages do require larger sizes.

I think it is an unnecessary burden on receiver implementors to be
required to receive 64K messages when 98% of all syslog messages are
likely to be <1K, and 99.9% will be under <2K, just to support those
unusual cases where messages may be as large as 32K or 70K. I dislike
this approach because if all receivers SHOULD support 64k messages,
that will tend to encourage implementors of message generators to make
their messages large and wordy rather than succinct, which could have
a negative impact on the primary use case for syslog. If we set the
required message support to be smaller, most implementors will keep
their messages smaller. 

I suggest that the required maximum size be more on the order of 2k,
4k, or 8k rather than 64k. This reduced receiver requirement should be
accommpanied by advice that some operational environments, such as
that specified in RFC3881, may require support for larger messages,
and reciever implementors MAY support larger messages or MAY make the
maximum message size a value that is configurable by the administrator
of the system, based on their operational needs.

This would keep most syslog messages small, yet accommodate special
operational needs such as health care security auditing.

David Harrington
[email-masked]

-----Original Message-----

David and all,

####
5.1  Message Length

Rainer

---

From: "Anton Okmianski" <[email-masked]>
Date: Thu, 14 Oct 2004 10:35:56 -0400

Rainer:

I agree with a SHOULD on 64k support on receivers and MAY on larger
sizes.     

I think we need to remove or change this sentence: "A receiver MUST be
able to accept messages up to and including 480 octets in length".  

1. Is this an absolute requirement?  We keep thinking about general
purpose receiver libraries.  But it does not have to be general
purpose.  If I have two specific applications with their own syslog
sender/receiver implementations (how hard is it to send/receive a udp
message) and they exchange syslog messages which are known not to
exceed 400 bytes, should my receiver be required to support 8k
messages? 

2. Why 480?  If this is intended to ensure basic interoperability,
then at least this should be a size greater than 1K (size previous
syslog RFC observed).  Maybe 4k or 8k.  At least it is a better
arbitrary number.  

So, I would support either removing the MUST requirement all together
or at least changing it to a more reasonable size. The latter approach
would be more limiting for implementations, but will insure
interoperability for general-purpose syslog libraries if this is the
overriding goal. 

Thanks,
Anton. 

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf 
> Of David B Harrington
> Sent: Thursday, October 14, 2004 8:53 AM
> To: 'Rainer Gerhards'; [email-masked]
> Subject: RE: [Syslog-sec] RE: Maximum message size 
> 
> Hi Rainer,
> 
> "For interoperability reasons, all receiver implementations 
> SHOULD be able to accept messages up to and including 65,535 
> octets in length."
> 
> Also, I think the last two sentences could be combined into 
> one - "If a receiver receives a message with a length larger 
> than 65,535 octets, or larger than it supports, the receiver 
> MAY discard the message or truncate the payload."
> 
> dbh
> 
> [mailto:[email-masked]] On Behalf Of 
> Rainer Gerhards
> I agree that 16 MB for a single message is far more than 
> actually should be used. Let's look a bit at the history of this:
> 
> Initially, we noticed that the 1024 octet limit for current 
> syslog is too short. I think there is still general agreement 
> on this. We had a good discussion about the size issue in 
> September 2003. A link to an archive is here:
> 
> The general issue is that 1K is too short for many 
> applications. For example, Marshall Glen mentions the 
> healthcare needs (RFC3881) for a larger message. As far as I 
> know, the IHE initiative currently standardizes on a maximum 
> XML stream size of 32K (which is then supposed to be 
> transmitted via syslog). I think a message size limit of 64K 
> would probably be suffcient for a long time to come (and 
> leaves some headroom for message content increases).
> 
> Besides that, I myself identified a need - in very rare 
> occasions - to send information larger than 64K. In extreme 
> cases, this information could be as large as one megabyte. 
> Again, these were cases that might happen once in a year. All 
> of this, of course, only if an operator
> *really* wants to submit that large information. So the 
> initial approach and suggestion was to introduce a way to 
> build "message groups". That would have been a way to split 
> "oversize messages" (as they were called
> then) into multiple syslog messages. They key to that concept 
> was that each single message could be kept within a 
> reasonable limit (it would have worked even with 1K max 
> message size). It would only have been a way to transmit very 
> large messages in those very seldom cases that they would 
> actually be used.
> 
> Some time later, we came to the conclusion that it should be 
> possible for a single syslog message to be of the largest 
> imaginable size. It was argued that fragmentation is a 
> transport issue and as such application-level "fragmentation" 
> (the message groups) would not do good. We agreed to that.
> 
> Even some time later, we needed to define a maximum message 
> size, just to limit it somewhere (if I recall correctly, that 
> was a transport requirement). We picked 16MB just as a number 
> that we thought would never have be reached and such be 
> sufficient for all imaginable time to come. As far as I was 
> concerend, that decision was driven by the fact that in the 
> past there were often limits that nobody expected to reach - 
> and yet they wer quickly reached.
> Som 16MB was picked more or less as "larger than anybody will 
> need anytime in the forseeable future". We did not even have 
> a big discussion on that limit.
> 
> I think this are the key points on how we reached this limit. 
> Now, thanks to your call and looking the whole situation, I 
> really think we've gone overboard. That 16MB limit is 
> something nobody intends to use. Even a lower - but high 
> limit - is extremely unlikely. I'd expect to see a 1MB 
> "message" in maybe 0.0001% of the cases - if at all (but I 
> Also, I have read your proposal to set a maxium size for the 
> message, but allow an implementation to go beyond that. Maybe 
> something like
> this:
> 
> ####
> 5.1  Message Length
> 
>    A receiver MUST be able to accept messages up to and 
> including 480 octets in length. 
> 
>    If a receiver receives a message with a length larger than 
> it supports, the receiver MAY 
> I've dropped the wording that transports MAY support only a 
> smaller max size - I think UDP was the most critical in this 
> regard and it obviously handles 64K. So I assume this clause 
> is not really needed if we go for 64K as the max size.
> 
> With that clause, we would limit the size for all interop 
> cases, but leave a window open so that the message length 
> could be extended if there is a real need AND the operator 
> and software vendor wants this.
> I assume nobody will implement this burden if there is no 
> market need to do so. But if it is, it clould at least be 
> implemented in a standards-compliant way.
> 
> If I look back at the original requirement - provide a way to 
> transmit very rare oversize messages, I think we can safely 
> implement this the bounds of the current proposals, even if 
> the size limit does not go beyond 64K. If somebody needs it, 
> this can be easily done via an experimental (or 
> vendor-specific) SD-ID. Again, I do not think somebody will 
> accept this burden without market need.
> 
> In the light of this, I, too, would recommend that we limit 
> the message size to 64K. I would prefer, however, that we 
> keep the maximum open as in the suggested wording above.
> 
> Rainer
> 

> > explicitly for the vendors of applications who want to use syslog
as
> a 

> > version header. If there is need to, we could always include that
in
> a

---

From: "David B Harrington" <[email-masked]>
Date: Thu, 14 Oct 2004 12:04:24 -0400

Keep in mind that the IETF is defining standards for cross-vendor
interoperability, not providing an umbrella for everybody's
implementation choices.
You can certainly build your implementation to support only 400 bytes
if that is what your vendor-specific usage demands; but you just
cannot claim compliance to the IETF standard for syslog.
I believe the 480 text should remain as the minimum size standard for
an IETF compliant vendor-interoperable implementation.

dbh 

-----Original Message-----
From: Anton Okmianski [mailto:[email-masked]] 
Sent: Thursday, October 14, 2004 10:36 AM
To: [email-masked]; 'Rainer Gerhards'; [email-masked]
Subject: RE: [Syslog-sec] RE: Maximum message size 

Rainer:

Thanks,
Anton. 

> [mailto:[email-masked]] On Behalf Of David
B 
> Harrington

> could be as large as one megabyte.
> Again, these were cases that might happen once in a year. All of
this, 
> of course, only if an operator

> Some time later, we came to the conclusion that it should be
possible 

> application-level "fragmentation"

> Now, thanks to your call and looking the whole situation, I really 
> think we've gone overboard. That 16MB limit is something nobody 
> intends to use. Even a lower - but high limit - is extremely
unlikely. 
> I'd expect to see a 1MB "message" in maybe 0.0001% of the cases - if

> at all (but I expect to see e.g. 70KB "message" in maybe 0.001% of
the 
> cases).
message, 

> If I look back at the original requirement - provide a way to
transmit 
> experimental (or
> vendor-specific) SD-ID. Again, I do not think somebody will accept 
> this burden without market need.
> 

as

in