Subject: RE: [Syslog-sec] Maximum message size

From: Marshall Glen <[email-masked]>
Date: Wed, 13 Oct 2004 12:56:19 -0400

While I'd agree that a 16M octet size is extreme, we do need to accommodate
larger syslog messages.  See RFC3881 for an example.  

Communication of healthcare application security audit data to a central
repository is the key use case for RFC3881.  The end-users are healthcare
security and privacy officers, and they will need aggregated data from
multiple IT-enable healthcare systems to assess security conformance and
track patterns of inappropriate data access.  The design of an XML schema to
meet the end-user information needs leads to a message size on the order of
a few kilobytes.  It needs to be sent reliably and securely to a central
repository for after-the-fact analysis.

Best regards,
Glen Marshall

-----Original Message-----
From: David B Harrington [mailto:[email-masked]]
Sent: Wednesday, October 13, 2004 11:10 AM
To: 'Rainer Gerhards'; 'Anton Okmianski'; [email-masked]
Subject: [Syslog-sec] Maximum message size 

Hi,

I am concerned that we are in danger of promoting uses of syslog that
will defeat its current usefulness as a human-readable log of system
activity.

Do network operators really believe the 16,777,216 octet size is a
needed improvement to syslog, or are we designing this size in
explicitly for the vendors of applications who want to use syslog as a
programmatic interface? I believe Syslog should be designed to meet
the needs of operators. Most of the discussion ssems to be from
vendors of syslog receivers or applications. Other protocols such as
FTP might be better used for such a specialty use case. 

Does allowing messages of 16,777,216 octets to be accumulated within
the system log make it harder to use some widely-available minimal
text editors, like Notepad, to view the system log? Will having huge
application-specific messages in the system log make it harder for an
operator to locate useful troubleshooting messages within the system
log? Can the information in such a huge message fit within an 80x25
screen, when an operator is troubleshooting network problems and needs
to use a bare-bones terminal attached to the serial port of a device
to view the system log? 

Syslog was designing to be an operator's tool, and Syslog should be
designed to meet the needs of operators.  Will allowing messages of
this size negatively impact its usefulness in the primary use case?  

What do network **operators** think about the need for messages of
this size? Have we deliberately asked them their opinion on this by,
for example, taking a survey or going to NANOG to ask them?

I don't think we should support such large messages in the system
logging protocol.

David Harrington
[email-masked]

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Rainer
Gerhards
Sent: Wednesday, October 13, 2004 5:17 AM
To: Anton Okmianski; [email-masked]
Subject: RE: [Syslog-sec] UDP message size issue proposal

Anton,

I agree to your conclusion.

Although I am the one who always voted for larger sizes, the
discussion has shown that this yields to unnecessary complexity. I can
satisfy my needs with a vendor-extension structured data element,
which I think shows the flexibility of the new drafts.

So I support your move. I would tend to even remove the transport
version header. If there is need to, we could always include that in a
later release (e.g. "v1", which would make it clearly distingshable
from the current frame format). I see little need to do so, though.

Regarding -protocol, I think we should still keep an upper limit in
the spec. Even I don't see any reason why a syslog message should go
above 16MB. So for -protocol, I propose the following new text:

####
5.1  Message Length

   The maximum length of any syslog message is 16,777,216 octets.  Any
   receiver receiving a larger message MUST discard the message.  A
   diagnostic message SHOULD be logged in this case.

   A receiver MUST be able to accept messages that are 480 octets (or
   less) in length.  A receiver SHOULD be able to accept messages that
   are 65,535 octets (or less) in length.  It is RECOMMENDED that
   receivers be able to accept messages up to the maximum message
length
   of 16,777,216 octets.

   If a receiver receives a message within the maximum length, but
with
   a length larger than it handles, the receiver MAY discard or
truncate
   it.

   A transport mapping MAY define a maximum length below the one
   specified here.  Each transport mapping MUST support a maximum size
   of 480 octets or more.
####

If there is agreement on this, I can post a new version of -protocol.
That version will also include some changes made thanks to Andrew Ross
comments (sent via private mail). I have finished this version. It is
available for immediate posting, but I would like to wait for
agreement on the size issue (at least if that can be reached quickly).

Rainer

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Anton 
> Okmianski
> Sent: Tuesday, October 12, 2004 11:25 PM
> To: [email-masked]
> Subject: [Syslog-sec] UDP message size issue proposal
> 
> Hi!
> 
> Sorry for a long delay on this issue - I was on a 2 week vacation.
I 
> have spoken with a number of TCP/UDP/IP experts regarding the sizing

> issue.  I am ready to propose the following changes:
> 
> 1. Syslog-protocol will state that the max message will be defined
by 
> the transport layer.
> 
> 2. Syslog-transport-udp will support messages up to maximum UDP 
> datagram size of 64K. UDP is a very bad choice for large message 
> transmissions, so it does not make sense for us to stretch it by 
> adding our own fragmentation without other transmission control 
> features such as found in TCP.
> 
> 3. Syslog-transport-udp will rely on IP fragmentation and we will
get 
> rid of "proprietary" fragmentation which was designed to handle 
> messages over 64K and deal with various
> non-compliant network hosts.    
> 
> 4. Syslog-transport-udp will recommend sending messages within the 
> boundaries of prevalent MTU size on a given network to be safe. It 
> will recommend Ethernet's 1500 bytes less headers and will draw
reader 
> attention to the minimum MTU size hosts on the network are required
to 
> support for
> IPv6 (576 bytes) and IPv6 (1280 bytes).   
> 
> 5. Path MTU discovery may not work robustly and some TCP/IP stacks
may 
> not support UDP packets of full 64K size and truncate them.  We will

> mention this and bite this bullet.
> We should not restrict the protocol due to incompliant
implementations 
> because it is a moving target and penalizes compliant
implementations 
> with extra overhead. The above size recommendation would partially 
> deal with this, but leave the
> final choice up to the administrator.      
> 
> 6. We will get rid of most syslog transport headers for UDP as they 
> will no longer be needed.  The only thing that will be left is the 
> transport protocol version prefixed to every syslog message.  Should

> we even bother with that?
> 
> This is a major change to the syslog-transport-udp.  I'd like to get

> positive feedback before I proceed with this update.
> The best part is that if we all agree on the above, the next draft 
> will be 1/3 of the size -- easier read for you. :)
> 
> Thanks,
> Anton. 
>     
> 
> _______________________________________________
> Syslog-sec mailing list
> [email-masked]
> http://www.employees.org/mailman/listinfo/syslog-sec
> 
_______________________________________________
Syslog-sec mailing list
[email-masked]
http://www.employees.org/mailman/listinfo/syslog-sec

_______________________________________________
Syslog-sec mailing list
[email-masked]
http://www.employees.org/mailman/listinfo/syslog-sec

-------------------------------------------------------------------------------
This message and any included attachments are from Siemens Medical Solutions 
USA, Inc. and are intended only for the addressee(s).  
The information contained herein may include trade secrets or privileged or 
otherwise confidential information.  Unauthorized review, forwarding, printing, 
copying, distributing, or using such information is strictly prohibited and may 
be unlawful.  If you received this message in error, or have reason to believe 
you are not authorized to receive it, please promptly delete this message and 
notify the sender by e-mail with a copy to [email-masked] 

Thank you
_______________________________________________
Syslog-sec mailing list
[email-masked]
http://www.employees.org/mailman/listinfo/syslog-sec

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 14 Oct 2004 18:32:35 +0200

David,

I see your point. I have no objections in limiting the message size to
2K as long as we keep the door open for senders/receivers supporting
larger message sizes. Then the market can decide ... and if there is
need for larger sizes, an application can implement it and still claim
compliance to the syslog standard. For example, it might be a
competitive advantage to support 32K messages, so that a product can be
used for the healthcare environment. As such, a vendor might want to
implement it. As long as we do not disallow this, the small size is not
a problem. And, yes, I agree it will encourage implementors to create
short messages.

So I am now proposing the following text:

####
5.1  Message Length

A receiver MUST be able to accept messages up to and including 480
octets in length. For interoperability reasons, all receiver
implementations 
SHOULD be able to accept messages up to and including 2,048 octets 
in length.

If a receiver receives a message with a length larger than 2,048 octets,
or larger than it supports, the receiver MAY discard the message or
truncate the payload.
####
 
Rainer

> Sent: Thursday, October 14, 2004 3:21 PM
> To: Rainer Gerhards; [email-masked]
> Subject: RE: [Syslog-sec] RE: Maximum message size 
> 
> Hi,
> 
> I have concerns about the requirements for maximum message size, and
> would like to propose an alternate approach.
> 
> Most syslog messages currently are less than 1K in size. There is
> consensus that 1K is typically acceptable, but it is a bit restrictive
> in some circunstances and the size should be increased. I agree with
> that consensus. I disagree that the size needs to be increased 64
> times to accommodate normal usage, and I recognize that some valid,
> but abnormal, usages do require larger sizes.
> 
> I think it is an unnecessary burden on receiver implementors to be
> required to receive 64K messages when 98% of all syslog messages are
> likely to be <1K, and 99.9% will be under <2K, just to support those
> unusual cases where messages may be as large as 32K or 70K. I dislike
> this approach because if all receivers SHOULD support 64k messages,
> that will tend to encourage implementors of message generators to make
> their messages large and wordy rather than succinct, which could have
> a negative impact on the primary use case for syslog. If we set the
> required message support to be smaller, most implementors will keep
> their messages smaller. 
> 
> I suggest that the required maximum size be more on the order of 2k,
> 4k, or 8k rather than 64k. This reduced receiver requirement should be
> accommpanied by advice that some operational environments, such as
> that specified in RFC3881, may require support for larger messages,
> and reciever implementors MAY support larger messages or MAY make the
> maximum message size a value that is configurable by the administrator
> of the system, based on their operational needs.
> 
> This would keep most syslog messages small, yet accommodate special
> operational needs such as health care security auditing.
> 
> David Harrington
> [email-masked]
> 
> 
> Sent: Thursday, October 14, 2004 7:34 AM
> Subject: [Syslog-sec] RE: Maximum message size 
> 
> David and all,
> 
> many thanks for all the good comments.
> 
> I agree that 16 MB for a single message is far more than actually
> should be used. Let's look a bit at the history of this:
> 
> Initially, we noticed that the 1024 octet limit for current syslog is
> too short. I think there is still general agreement on this. We had a
> good discussion about the size issue in September 2003. A link to an
> archive is here:
> 
> http://www.syslog.cc/ietf/autoarc/msg00855.html
> 
> The general issue is that 1K is too short for many applications. For
> example, Marshall Glen mentions the healthcare needs (RFC3881) for a
> larger message. As far as I know, the IHE initiative currently
> standardizes on a maximum XML stream size of 32K (which is then
> supposed to be transmitted via syslog). I think a message size limit
> of 64K would probably be suffcient for a long time to come (and leaves
> some headroom for message content increases).
> 
> Besides that, I myself identified a need - in very rare occasions - to
> send information larger than 64K. In extreme cases, this information
> could be as large as one megabyte. Again, these were cases that might
> happen once in a year. All of this, of course, only if an operator
> *really* wants to submit that large information. So the initial
> approach and suggestion was to introduce a way to build "message
> groups". That would have been a way to split "oversize messages" (as
> they were called
> then) into multiple syslog messages. They key to that concept was that
> each single message could be kept within a reasonable limit (it would
> have worked even with 1K max message size). It would only have been a
> way to transmit very large messages in those very seldom cases that
> they would actually be used.
> 
> Some time later, we came to the conclusion that it should be possible
> for a single syslog message to be of the largest imaginable size. It
> was argued that fragmentation is a transport issue and as such
> application-level "fragmentation" (the message groups) would not do
> good. We agreed to that.
> 
> Even some time later, we needed to define a maximum message size, just
> to limit it somewhere (if I recall correctly, that was a transport
> requirement). We picked 16MB just as a number that we thought would
> never have be reached and such be sufficient for all imaginable time
> to come. As far as I was concerend, that decision was driven by the
> fact that in the past there were often limits that nobody expected to
> reach - and yet they wer quickly reached.
> Som 16MB was picked more or less as "larger than anybody will need
> anytime in the forseeable future". We did not even have a big
> discussion on that limit.
> 
> I think this are the key points on how we reached this limit. Now,
> thanks to your call and looking the whole situation, I really think
> we've gone overboard. That 16MB limit is something nobody intends to
> use. Even a lower - but high limit - is extremely unlikely. I'd expect
> to see a 1MB "message" in maybe 0.0001% of the cases - if at all (but
> I expect to see e.g. 70KB "message" in maybe 0.001% of the cases).
> 
> Also, I have read your proposal to set a maxium size for the message,
> but allow an implementation to go beyond that. Maybe something like
> this:
> 
> ####
> 5.1  Message Length
> 
>    A receiver MUST be able to accept messages up to and including 480
> octets in length. 
> 
>    A receiver SHOULD be able to accept messages up to and including
> 65,535 octets in length. 
>    If a receiver receives a message with a length larger than it
> supports, the receiver MAY 
>    discard the message or truncate the payload.
> 
>    A receiver MAY accept messages larger than 65,535 octets.
> ####
> 
> I've dropped the wording that transports MAY support only a smaller
> max size - I think UDP was the most critical in this regard and it
> obviously handles 64K. So I assume this clause is not really needed if
> we go for 64K as the max size.
> 
> With that clause, we would limit the size for all interop cases, but
> leave a window open so that the message length could be extended if
> there is a real need AND the operator and software vendor wants this.
> I assume nobody will implement this burden if there is no market need
> to do so. But if it is, it clould at least be implemented in a
> standards-compliant way.
> 
> If I look back at the original requirement - provide a way to transmit
> very rare oversize messages, I think we can safely implement this the
> bounds of the current proposals, even if the size limit does not go
> beyond 64K. If somebody needs it, this can be easily done via an
> experimental (or vendor-specific) SD-ID. Again, I do not think
> somebody will accept this burden without market need.
> 
> In the light of this, I, too, would recommend that we limit the
> message size to 64K. I would prefer, however, that we keep the maximum
> open as in the suggested wording above.
> 
> Rainer
> 
> > Sent: Wednesday, October 13, 2004 5:10 PM
> > To: Rainer Gerhards; 'Anton Okmianski'; [email-masked]
> > Subject: Maximum message size
> > 
> > Hi,
> >

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 21 Oct 2004 17:54:52 +0200

Hi all,

I share David's concerns. I still think limiting to 480 octets is
necessary to support the broken-network use case that David outlined.
Actually, this is one use case, but I think a very important one. For
these type of alert messages, 480 octets is far then enough. Traditional
syslog messages, at least in UNIX environments, are typically even below
100 octets.

There is another use case, which is transmittal of logging information
during normal operation (like in healthcare and other auditing
environments). Here, this size is often too small. But by allowing an
application to support sending/receiving larger message sizes, we
support this scenario, too.

I think we have created a good compromise by the wording below:

####
5.1  Message Length

Actively participating in the discussion were Anton and David so far. As
Anton said he can live with this way, too, I think this is the current
rough concensus.

If anybody violently OBJECTS the given wording, please make yourself
heared NOW. 

Otherwise, I will see that I can change the draft tomorrow so that it
can (hopefully) be submitted before the meeting cutoff date.

Rainer

> [mailto:[email-masked]] On Behalf Of 
> David B Harrington
> Sent: Thursday, October 21, 2004 5:11 PM
> To: 'Anton Okmianski'; 'Chris Lonvick'; [email-masked]
> The reason SNMP has a 480 minimum-maximum has to do with difficult
> delivery problems in a broken network. SNMP uses UDP and has this 480
> restriction deliberately to deliberately avoid session overhead and
> message fragmentation. In a network being troubleshot, the likelihood
> of getting one single-packet message delivered successfully is higher
> than getting three message fragments delivered successfully, so using
> a 1K minimum-maximum may prevent the operator from getting some
> critical information about the problem, whereas using a 480
> minimum-maximum might get that information to the operator.
> 
> Which again raises the issue of the original use case for syslog. Is
> syslog meant to be a protocol useful for troubleshooting devices over
> a network? If the WG decides this is no longer an important use case,
> then I suggest the WG should have a long hard discussion with
> operators to see if they agree with this decision.
> 
> If syslog is to remain a primary tool for operators to troubleshoot
> broekn networks, then it should be designed with broken-network
> delivery problems in mind. It should avoid encouraging implementors to
> use a minimum-maximum messages size that would require fragmentation.
> 
> dbh
> co-chair SNMPv3 WG, concluded
> 
> Sent: Thursday, October 21, 2004 10:24 AM
> To: 'Chris Lonvick'; [email-masked]
> Rainer, Chris, all:
> My vote is for -protocol to say that any syslog *transport* MUST
> support transferring messages of at least 1Kb (or 2 Kb) and recommend
> that they support 64Kb.  Udp-transport will, in turn, recommend that
> applications don't send message with sizes which exceed the MTU on a
> given network to avoid fragmentation and provide Ethernet MTU (less
> headers) as a guide. 
> 
> So, if somebody' network has the old MTUs of 576 bytes, we would still
> handle it, they will just need to set-up their TCP-IP stack with
> appropriate MTU (assuming MTU Discovery does not work) and there will
> be fragmentation for messages greater than this size.   
> 
> We could also say that *receivers* SHOULD support messages of at least
> 1Kb (or 2Kb).  
> I would not set any MUST requirements on the receivers.    Note the
> difference between requirements on receivers and transport protocol.
> 
> Rainer, please get the consensus on this on the list and let's move on
> one way or the other.  I will be happy with whatever the majority
> consensus is.
> 
> Thanks,
> Anton. 
> 
> 
> 
> > From: [email-masked]
> > [mailto:[email-masked]] On Behalf Of Chris
> 
> > Lonvick
> > Sent: Thursday, October 21, 2004 8:44 AM
> > Hi All,
> > Do we have rough consensus on this?  We can still request a slot at 
> > the upcoming IETF to discuss this.
> > 
> > Thanks,
> > Chris
> > 
> > On Mon, 18 Oct 2004, Rainer Gerhards wrote:
> > 
> > > Anton:
> > >
> > > I understand the source of the 480 byte limit you outlined in 
> > > -transport. Even though we have now decided that we can use UDP 
> > > fragementation, I still think the fact that 480 is the only safe 
> > > number still persists.
> > >
> > > So I do not see any value in requiring an application to support 
> > > messages of up to 1K when we know this might not work. I agree, it
> 
> > > looks like it has worked (at least most of the time) so
> > far, but why
> > > introduce a problem if there is no need to? Maybe we had no
> > issues in
> > > the past because messages were most often below 480 octets.
> > >
> > > Also, I do not think that we can say the minimum to be
> > supported size
> > > is 1K in protocol and then allow a transport mapping to specify 
> > > something lower. If it is 1K, then it MUST be 1K in all
> > cases. Else I
> > > can not see how we will be able to ensure interoperability. I am
> of 
> > > the strong opinion we need to know a maximum size that is
> > guaranteed
> > > to be received.
> > >
> > > I do not see the value in raising the minimum to be
> > supported size to
> > > 1K. Maybe I am overlooking the obvious, but I think we
> deliberatley 
> > > introduce problems if we go above that.
> > >
> > > And, yes, I am refering to IPv4 only, because I am looking at the 
> > > weakest transport (which, in this case, is also the most
> > commonly used).
> > >
> > > Rainer
> > >
> > > > From: Anton Okmianski [mailto:[email-masked]]
> > > > Sent: Thursday, October 14, 2004 7:54 PM
> > > > To: Rainer Gerhards; [email-masked];
> [email-masked]
> > > > Rainer:
> > > > The ~480 byte limit was on the size of packet, not datagram or 
> > > > message which can be fragmented into multiple packets.
> > Besides, for
> > > > IPv6 it was ~1160.  In order to avoid requiring people to
> > implement
> > > > "proprietary" transport fragmentation we did say that 
> > > > implementations must only support messages of size which does
> not 
> > > > require this fragmentation.  But we don't do that fragmentation 
> > > > anymore and rely on IP fragmentation.
> > > >
> > > > If the intention of this MUST requirement is a baseline 
> > > > interoperability and we know that existing applications may be 
> > > > sending messages as large as 1K, then it seems to me the minimum
> 
> > > > required support should be at least that big.
> > > >
> > > > The transport may provide its own recommendations.  For
> > example, udp
> > > > transport will *recommend* that messages do not exceed
> > the smallest
> > > > MTU size on the network, which could be smaller or
> > larger, but that
> > > > will be just a recommendation.  For TCP, it is not as big of an 
> > > > issue, for example, because it handles loss of segments.  I
> don't 
> > > > think such transport layer requirements should be in the
> protocol 
> > > > which describes the content of the messages.
> > > >
> > > > Cheers,
> > > >
> > > > Anton.
> > > >
> > > > > From: Rainer Gerhards [mailto:[email-masked]]
> > > > > Sent: Thursday, October 14, 2004 11:00 AM
> > > > > To: Anton Okmianski; [email-masked];
> > [email-masked]
> > > > > Hello Anton:
> > > > > the 480 octet limit stems back to your work and proposal in 
> > > > > -transport-udp. I thought you found out that more than
> > 480 bytes
> > > > > can not be transmitted "reliably" (whatever this means for
> > > > UDP).
> > > > >
> > > > > Of course, I can remove this and we could simply state that 
> > > > > senders send data and receivers receive data and that
> > the receiver
> > > > > MAY receive any size or not. But wouldn't it make sense to
> have 
> > > > > one size that I can count on to be received?
> > > > >
> > > > > The intension of the 480 octets was to provide a known message
> 
> > > > > size that every implementation actually MUST support
> > > > > - so if an applications intends to send a message that MUST be
> 
> > > > > received fully, that application knows a maximum size.
> > > > >
> > > > > If you mean that with the upcoming version of
> > -transport-udp the
> > > > > 480 octet can safely be replaced by 1K or 2K then we of
> > course can
> > > > > change this.
> > > > >
> > > > > I, however, think that a minimal burden should be placed on an
> 
> > > > > implementation. I would tend to call a syslog receiver that is
> 
> > > > > only capable of receiving 400 octets to be
> > non-compliant. But, OK,
> > > > > maybe we should say this is OK. So what limit is it
> > then? Is 200
> > > > > octets compliant? 100? 50?
> > > > > Whatever it is, I think we should require one size... 
> > If we do not
> > > > > require this, one could argue writing/parsing a correct
> > header is
> > > > > also too much of a burden for a given scenario...
> > > > >
> > > > > Rainer
> > > > >
> > > > > > From: Anton Okmianski [mailto:[email-masked]]
> > > > > > Sent: Thursday, October 14, 2004 4:36 PM
> > > > > > To: [email-masked]; Rainer Gerhards; 
> > > > > > [email-masked]
> > > > > > Rainer:
> > > > > > I agree with a SHOULD on 64k support on receivers and MAY on
> > > > larger
> > > > > > sizes.
> > > > > >
> > > > > > I think we need to remove or change this sentence: "A
> > > > > receiver MUST be
> > > > > > able to accept messages up to and including 480
> > octets in length".
> > > > > >
> > > > > > 1. Is this an absolute requirement?  We keep thinking about
> > > > general
> > > > > > purpose receiver libraries.  But it does not have to
> > be general
> > > > > > purpose.  If I have two specific applications with their own
> > > > syslog
> > > > > > sender/receiver implementations (how hard is it to
> > > > > send/receive a udp
> > > > > > message) and they exchange syslog messages which are
> > known not
> > > > > > to exceed 400 bytes, should my receiver be required
> > to support
> > > > > > 8k messages?
> > > > > >
> > > > > > 2. Why 480?  If this is intended to ensure basic 
> > > > > > interoperability,
> > > >
> > > > > > then at least this should be a size greater than 1K (size 
> > > > > > previous
> > > >
> > > > > > syslog RFC observed).  Maybe 4k or 8k.  At least it
> > is a better
> > > > > > arbitrary number.
> > > > > >
> > > > > > So, I would support either removing the MUST requirement
> > > > > all together
> > > > > > or at least changing it to a more reasonable size. The
> > > > > latter approach
> > > > > > would be more limiting for implementations, but will insure 
> > > > > > interoperability for general-purpose syslog libraries if
> > > > > this is the
> > > > > > overriding goal.
> > > > > >
> > > > > > Thanks,
> > > > > > Anton.
> > > > > >
> > > > > > > From: [email-masked]
> > > > > > > [mailto:[email-masked]] On
> > > > > Behalf Of David
> > > > > > > B Harrington
> > > > > > > Sent: Thursday, October 14, 2004 8:53 AM
> > > > > > > To: 'Rainer Gerhards'; [email-masked]
> > > > > > > Hi Rainer,
> > > > > > > Thanks for making the changes.
> > > > > > >
> > > > > > > I'd like to make one point a little more strenuously:
> > > > > > > "For interoperability reasons, all receiver
> > > > > implementations SHOULD
> > > > > > > be able to accept messages up to and including
> > 65,535 octets
> > > > > > > in length."
> > > > > > >
> > > > > > > Also, I think the last two sentences could be combined
> into 
> > > > > > > one
> > > > -
> > > > > > > "If a receiver receives a message with a length larger
> > > > > than 65,535
> > > > > > > octets, or larger than it supports, the receiver
> > MAY discard
> > > > > > > the
> > > >
> > > > > > > message or truncate the payload."
> > > > > > >
> > > > > > > dbh
> > > > > > >
> > > > > > > Initially, we noticed that the 1024 octet limit for
> > > > > current syslog
> > > > > > > is too short. I think there is still general agreement on
> > > > > this. We
> > > > > > > had a good discussion about the size issue in September
> > > > > 2003. A link
> > > > > > > to an archive is here:
> > > > > > >
> > > > > > > standardizes on a maximum XML stream size of 32K (which is
> 
> > > > > > > then supposed to be transmitted via syslog). I 
> > think a message
> > > > > size limit
> > > > > > > of 64K would probably be suffcient for a long time to come
> 
> > > > > > > (and leaves some headroom for message content increases).
> > > > > > >
> > > > > > > Besides that, I myself identified a need - in very rare
> > > > > occasions -
> > > > > > > to send information larger than 64K. In extreme cases,
> this 
> > > > > > > information could be as large as one megabyte.
> > > > > > > Again, these were cases that might happen once in a 
> > year. All 
> > > > > > > of
> > > >
> > > > > > > this, of course, only if an operator
> > > > > > > *really* wants to submit that large information. So the 
> > > > > > > initial approach and suggestion was to introduce a way to 
> > > > > > > build "message
> > > >
> > > > > > > then) into multiple syslog messages. They key to 
> > that concept
> > > > was
> > > > > > > that each single message could be kept within a
> > > > > reasonable limit (it
> > > > > > > would have worked even with 1K max message size). It
> > > > > would only have
> > > > > > > been a way to transmit very large messages in those very 
> > > > > > > seldom cases that they would actually be used.
> > > > > > >
> > > > > > > Some time later, we came to the conclusion that it 
> > should be 
> > > > > > > possible for a single syslog message to be of the largest
> > > > > imaginable
> > > > > > > size. It was argued that fragmentation is a transport
> > > > > issue and as
> > > > > > > such application-level "fragmentation"
> > > > > > > (the message groups) would not do good. We agreed to that.
> > > > > > >
> > > > > > > Even some time later, we needed to define a maximum
> message
> > > > size,
> > > > > > > just to limit it somewhere (if I recall correctly, 
> > that was a 
> > > > > > > transport requirement). We picked 16MB just as a 
> > number that 
> > > > > > > we thought would never have be reached and such be
> > > > > sufficient for all
> > > > > > > imaginable time to come. As far as I was concerend, that
> > > > decision
> > > > > > > was driven by the fact that in the past there were often
> > > > > limits that
> > > > > > > nobody expected to reach - and yet they wer quickly
> reached.
> > > > > > > I think this are the key points on how we reached 
> > this limit.
> > > > > > > Now, thanks to your call and looking the whole situation,
> > > > > I really
> > > > > > > think we've gone overboard. That 16MB limit is something 
> > > > > > > nobody intends to use. Even a lower - but high limit - is 
> > > > > > > extremely unlikely. I'd expect to see a 1MB 
> > "message" in maybe
> > > > > 0.0001% of the
> > > > > > > cases - if at all (but I expect to see e.g. 70KB
> > > > > "message" in maybe
> > > > > > > 0.001% of the cases).
> > > > > > >
> > > > > > > Also, I have read your proposal to set a maxium 
> > size for the 
> > > > > > > message, but allow an implementation to go beyond 
> > that. Maybe 
> > > > > > > something like
> > > > > > > this:
> > > > > > >
> > > > > > > ####
> > > > > > > 5.1  Message Length
> > > > > > >
> > > > > > >    A receiver MUST be able to accept messages up to and
> > > > including
> > > > > > > 480 octets in length.
> > > > > > >
> > > > > > >    If a receiver receives a message with a length 
> > larger than 
> > > > > > > it
> > > >
> > > > > > > obviously handles 64K. So I assume this clause is not
> > > > > really needed
> > > > > > > if we go for 64K as the max size.
> > > > > > >
> > > > > > > I assume nobody will implement this burden if there is no 
> > > > > > > market
> > > >
> > > > > > > need to do so. But if it is, it clould at least be
> > > > > implemented in a
> > > > > > > standards-compliant way.
> > > > > > >
> > > > > > > If I look back at the original requirement - 
> > provide a way to 
> > > > > > > transmit very rare oversize messages, I think we can
> safely 
> > > > > > > implement this the bounds of the current proposals, even
> > > > > if the size
> > > > > > > limit does not go beyond 64K. If somebody needs it, 
> > this can 
> > > > > > > be easily done via an experimental (or
> > > > > > > vendor-specific) SD-ID. Again, I do not think somebody
> > > > > will accept
> > > > > > > this burden without market need.
> > > > > > >
> > > > > > > In the light of this, I, too, would recommend that we
> limit 
> > > > > > > the message size to 64K. I would prefer, however, 
> > that we keep 
> > > > > > > the maximum open as in the suggested wording above.
> > > > > > >
> > > > > > > Rainer
> > > > > > >
> > > > > > > > I am concerned that we are in danger of promoting uses
> of
> > > > syslog
> > > > > > > that
> > > > > > > > will defeat its current usefulness as a 
> > human-readable log 
> > > > > > > > of
> > > > > > system
> > > > > > >
> > > > > > > > activity.
> > > > > > > >
> > > > > > > > needed improvement to syslog, or are we designing 
> > this size 
> > > > > > > > in
> > > >
> > > > > > > > explicitly for the vendors of applications who want to
> > > > > use syslog
> > > > > > as
> > > > > > > a
> > > > > > > > programmatic interface? I believe Syslog should 
> > be designed 
> > > > > > > > to
> > > > > > meet
> > > > > > > > the needs of operators. Most of the discussion ssems to
> be
> > > > from
> > > > > > > > vendors of syslog receivers or applications. Other
> > > > > protocols such
> > > > > > as
> > > > > > >
> > > > > > > > Does allowing messages of 16,777,216 octets to be 
> > > > > > > > accumulated
> > > > > > within
> > > > > > >
> > > > > > > > text editors, like Notepad, to view the system log? Will
> > > > having
> > > > > > huge
> > > > > > >
> > > > > > > > application-specific messages in the system log make it
> > > > > harder for
> > > > > > > an
> > > > > > > > operator to locate useful troubleshooting messages
> within 
> > > > > > > > the
> > > > > > system
> > > > > > >
> > > > > > > > log? Can the information in such a huge message 
> > fit within 
> > > > > > > > an
> > > > > > 80x25
> > > > > > > > screen, when an operator is troubleshooting 
> > network problems
> > > > and
> > > > > > > needs
> > > > > > > > to use a bare-bones terminal attached to the 
> > serial port of 
> > > > > > > > a
> > > > > > device
> > > > > > >
> > > > > > > > to view the system log?
> > > > > > > >
> > > > > > > > Syslog was designing to be an operator's tool, and
> Syslog
> > > > should
> > > > > > be
> > > > > > > > designed to meet the needs of operators.  Will allowing
> > > > messages
> > > > > > of
> > > > > > > > this size negatively impact its usefulness in the
> primary 
> > > > > > > > use
> > > > > > case?
> > > > > > > >
> > > > > > > > this size? Have we deliberately asked them their 
> > opinion on
> > > > this
> > > > > > by,
> > > > > > >
> > > > > > > > discussion has shown that this yields to unnecessary
> > > > > complexity. I
> > > > > > > can
> > > > > > > > version header. If there is need to, we could always
> > > > > include that
> > > > > > in
> > > > > > > a
> > > > > > > > from the current frame format). I see little need 
> > to do so,
> > > > > > though.
> > > > > > > >
> > > > > > > > Regarding -protocol, I think we should still keep an
> upper
> > > > limit
> > > > > > in
> > > > > > > > the spec. Even I don't see any reason why a syslog
> > > > > message should
> > > > > > go
> > > > > > >
> > > > > > > > above 16MB. So for -protocol, I propose the following
> new
> > > > text:
> > > > > > > >
> > > > > > > > ####
> > > > > > > > 5.1  Message Length
> > > > > > > >
> > > > > > > >    The maximum length of any syslog message is
> > > > > 16,777,216 octets.
> > > > > > > Any
> > > > > > > >    receiver receiving a larger message MUST discard the
> > > > message.
> > > > > > A
> > > > > > > >    A receiver MUST be able to accept messages that are
> > > > > 480 octets
> > > > > > > (or
> > > > > > > >    less) in length.  A receiver SHOULD be able to
> > > > > accept messages
> > > > > > > that
> > > > > > > >    a length larger than it handles, the receiver 
> > MAY discard
> > > > or
> > > > > > > > truncate
> > > > > > > >    it.
> > > > > > > >
> > > > > > > >    A transport mapping MAY define a maximum 
> > length below the
> > > > one
> > > > > > > >    specified here.  Each transport mapping MUST support
> > > > > a maximum
> > > > > > > size
> > > > > > > >    of 480 octets or more.
> > > > > > > > ####
> > > > > > > >
> > > > > > > > If there is agreement on this, I can post a new version
> of
> > > > > > > -protocol.
> > > > > > > > That version will also include some changes made thanks
> > > > > to Andrew
> > > > > > > Ross
> > > > > > > > comments (sent via private mail). I have finished this
> > > > > version. It
> > > > > > > is
> > > > > > > > available for immediate posting, but I would like to
> wait 
> > > > > > > > for agreement on the size issue (at least if that can be
> 
> > > > > > > > reached
> > > > > > > quickly).
> > > > > > > >
> > > > > > > > Rainer
> > > > > > > >
> > > > > > > > > Sorry for a long delay on this issue - I was on a 2
> week
> > > > > > vacation.
> > > > > > > > I
> > > > > > > > > have spoken with a number of TCP/UDP/IP experts 
> > regarding
> > > > the
> > > > > > > sizing
> > > > > > > >
> > > > > > > > > 1. Syslog-protocol will state that the max 
> > message will be
> > > > > > defined
> > > > > > > > by
> > > > > > > > > the transport layer.
> > > > > > > > >
> > > > > > > > > 3. Syslog-transport-udp will rely on IP 
> > fragmentation and 
> > > > > > > > > we
> > > > > > will
> > > > > > > > get
> > > > > > > > > messages over 64K and deal with various non-compliant 
> > > > > > > > > network hosts.
> > > > > > > > >
> > > > > > > > > 4. Syslog-transport-udp will recommend sending
> messages
> > > > within
> > > > > > the
> > > > > > >
> > > > > > > > > boundaries of prevalent MTU size on a given network
> > > > > to be safe.
> > > > > > It
> > > > > > >
> > > > > > > > > attention to the minimum MTU size hosts on the 
> > network are
> > > > > > > required
> > > > > > > > to
> > > > > > > > > support for
> > > > > > > > > 5. Path MTU discovery may not work robustly and some 
> > > > > > > > > TCP/IP
> > > > > > stacks
> > > > > > > > may
> > > > > > > > > not support UDP packets of full 64K size and truncate
> > > > > them.  We
> > > > > > > will
> > > > > > > >
> > > > > > > > > with extra overhead. The above size recommendation
> would
> > > > > > partially
> > > > > > >
> > > > > > > > > deal with this, but leave the final choice up to the 
> > > > > > > > > administrator.
> > > > > > > > >
> > > > > > > > > 6. We will get rid of most syslog transport headers
> for 
> > > > > > > > > UDP
> > > > as
> > > > > > > they
> > > > > > > > > will no longer be needed.  The only thing that will be
> 
> > > > > > > > > left
> > > > is
> > > > > > the
> > > > > > >
> > > > > > > > > transport protocol version prefixed to every 
> > syslog message.
> > > > > > > Should
> > > > > > > >
> > > > > > > > > This is a major change to the syslog-transport-udp.
> > > > > I'd like to
> > > > > > > get
> > > > > > > >
> > > > > > > > > The best part is that if we all agree on the above,
> the 
> > > > > > > > > next
> > > > > > draft
> > > > > > >