Subject: [Syslog] Syslog Protocol doubts

From: "Glenn M. Keeni" <[email-masked]>
Date: Fri, 05 Jan 2007 20:31:55 +0900

Hi,
   I have been trying to figure out the error conditions
that a syslog receiver will need to anticipate and the
corresponding actions that it is expected to take. I do
not see this clearly spelt out in the protocol document.
There are several MUST clauses, I understand that a
compliant syslog sender WILL always send messages that
meet the MUST clauses. But the document does not spell
out clearly what a compliant syslog receiver will do
when it gets a non-compliant message. Possible actions
could be:
        a. discard whole message
        b. discard non-compliant part ( assuming the non-
           compliant part can be isolated)
        c. rectify the non-compliance e.g.
            - truncate message: [this is mentioned in 6.1]
            - truncate the long-fields (software,
              swVersion etc.)
        d. implementation dependent

    Is this a problem ? I have listed the MUST conditions
in the attached document. My take is that we may have to
address each one of these. Or we can include a sweeping
statement like " non-compliant messages will be discarded"
or "the handling of non-compliant messages will be
implementation dependent".

   Glenn

Sec. 6.2     The character set used in the HEADER MUST be 
             seven-bit ASCII
Doubt:       What if it it isn't ? Will the receiver discard it ?

Sec. 6.2.1   The PRI part MUST have three, four, or five characters.. 
Doubt:       What the PRI part has two or six characters ? 
             Will the receiver discard it ?

Sec. 6.2.3   The "T" and "Z" characters in this syntax MUST be 
             upper case.
Doubt:       What if it is lower case ? Will the receiver discard it ?

Sec. 6.2.3   Usage of the "T" character is REQUIRED
Doubt:       What if it is absent ? Discard message ?

Sec. 6.2.3   Leap seconds MUST NOT be used
Doubt:       What if it is used? Discard message? 

Sec. 6.2.3.1 Example 4. Invalid timestamp
Doubt:       Will this message be discarded?

Sec. 6.2.4   If an IPv4 address is used, it MUST be in the format of 
             the dotted decimal notation as used in STD 13 [3]
Doubt:       What if it is not in the dotted decimal notation of 
             STD 13? 

Sec. 6.3     A receiver MAY ignore malformed STRUCTURED-DATA elements. 
Doubt:       But in sec. 6.4 Example 4 of a malformed STRUCTURED-DATA 
             element it says ... "the receiver MAY discard this 
             message".  

Sec. 6.4     If a sender encodes MSG in UTF-8, the string MUST start 
             with the Unicode byte order mask (BOM), which for UTF-8 
             is ABNF %xEF.BB.BF.
Doubt:       What if it doesn't start with a  BOM? Discard message ?

Sec. 7.1.1   If it does so, the value "1" MUST be used.  If the
             time zone information is in doubt, the value "0" MUST 
             be used.
Doubt:       What if the value "1" is used, inappropriately ? 
             Discard message ?

Sec. 7.1.2   If the original sender is time synchronized, the value 
             "1" MUST be used. If not, the value "0" MUST be used.
Doubt:       What if the neither 0 nor 1 is used? What if the usage 
             is inappropriate?

Sec. 7.1.3   If the value "0" is used for "isSynced", this parameter 
             MUST NOT be specified. 
Doubt:       What if it is specified ? Discard message ?

Sec. 7.2.2.  The "enterpriseId" parameter MUST be a 'SMI Network 
             Management Private Enterprise Code', maintained by IANA, 
             whose prefix is iso.org.dod.internet.private.enterprise 
             (1.3.6.1.4.1).
Doubt:       What if the enterpriseId doe not have prefix 1.3.6.1.4.1 ?

Sec. 7.2.3   The "software" parameter is a string. It MUST NOT be 
             longer than 48 characters.
Doubt:       What if it is longer than 48 characters ? Truncate ? 
             Discard ?

Sec. 7.2.4   The "swVersion" parameter is a string.  It MUST NOT be 
             longer than 32 characters.
Doubt:       What if it is longer than 32 characters ? Truncate ? 
             Discard ?

Sec. 7.3.2   the value MUST be represented as a decimal integer 
             (no decimal point) using only the characters "0", "1", 
             "2", "3", "4", "5", "6", "7", "8", and "9".
Doubt:       What if it not a decimal integer ? Discard message ?

Sec. 7.3.3   If it is specified, it MUST contain a two letter language 
             identifier as defined in ISO 639 [13].
Doubt:       What if the language identifier is not defined in ISO 
             639 [13]? Discard message? 
_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 5 Jan 2007 14:46:42 +0100

Hi Glen,

thanks for the message. Let me start on an overview level: if you look
at the evolution of the draft, you will see that earlier versions were
quite specific on what to do if the message was malformed. However,
based on dicussion, one after another of these rules were deleted. The
reason was always that MUSTs here are not actually needed to ensure
interoperability.

You can get a glimpse of this discussion by looking at

http://www.syslog.cc/ietf/why-indepth.html

This is a very old page. For more recent samples, you should consult the
mailing list archives. There are (too) plenty samples of this being
discussed to point to anything specific.

The bottom line behind the current draft is that we do not necessarily
specify what happens if the message is malformed. This is not vital for
interoperability. Also, implementors will provide different solutions
(most probably operator-configurable) to address real-world needs. For
example, we could specify that a message with leap seconds in it MUST be
discarded - but if the operator insists that he needs such a message, an
implementor will always create a way to process it.

We are specific on invalid UTF-8 sequences. This must not be
interpreted, because this is a big security issue. However, even than it
might be OK for an implementation to store the invalid UTF-8 sequence.

If that is consensus, we can add the statement "the handling of
non-compliant messages will be implementation dependent" which very
precisely describes the list consensus.

Rainer

> -----Original Message-----
> From: Glenn M. Keeni [mailto:[email-masked]]
> Sent: Friday, January 05, 2007 12:32 PM
> To: [email-masked]
> Subject: [Syslog] Syslog Protocol doubts
> 
> Hi,

---

From: "David Harrington" <[email-masked]>
Date: Fri, 5 Jan 2007 10:31:39 -0500

Or we could simply reiterate Postel's law:

"In general, an implementation must be conservative
  in its sending behavior, and liberal in its receiving behavior.
That
  is, it must be careful to send well-formed datagrams, but must
accept
  any datagram that it can interpret (e.g., not object to technical
  errors where the meaning is still clear)." [STD5]

dbh

> From: Rainer Gerhards [mailto:[email-masked]] 
> Sent: Friday, January 05, 2007 8:47 AM
> To: Glenn M. Keeni; [email-masked]
> Subject: RE: [Syslog] Syslog Protocol doubts
> 
> Hi Glen,
>

---

From: "tom.petch" <[email-masked]>
Date: Fri, 5 Jan 2007 18:47:05 +0100

<inline>
Tom Petch

----- Original Message -----
From: "Rainer Gerhards" <[email-masked]>
To: "Glenn M. Keeni" <[email-masked]>; <[email-masked]>
Sent: Friday, January 05, 2007 2:46 PM
Subject: RE: [Syslog] Syslog Protocol doubts

Hi Glen,

<tp>

I take a somewhat different view.  To quote RFC2119,

"Imperatives of the type defined in this memo must be used with care
   and sparingly.  In particular, they MUST only be used where it is
   actually required for interoperation or to limit behavior which has
   potential for causing harm (e.g., limiting retransmisssions)."

So if you receive a packet that breaks a MUST, you MUST discard it.

If it breaks a SHOULD, then you MAY accept it, in fact I would say you ought to
on the principle of being liberal in what you accept.

<tom Petch>

<snip>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 5 Jan 2007 21:12:31 +0100

Hi Tom,

I think this is an extreme position. The reason is that if I take this
to its ultimate end, we would probably end up without any MUST in the
draft. For example, should we just say the date SHOULD be given in RFC
3339 format ... But leave it open to any implementor what he intends to
do? I think a MUST is justified here. But on the other hand, I can NOT
force ("MUST") an implementation to discard an otherwise useful message
just because the date is in the wrong format. From the protocol
perspective, this might be the right thing to do (in trying to punish
the incompliant implementation), but from a usabulity point of view, the
administrator needs to have this capability. In a closed-source
environment, that means marketing will force an implementor to support
it. In open-source, the administrator will simply modify the
implementation himself. In either way, the "MUST discard" will not
survive. So is it really smart to mandate something in a standard that
we can foresee to be ignored (and that for good reason)? However, we
need to set same baseline and these are the format-MUSTs. If we change
them all to SHOULD (to cover a myriad of real-world cases), we do not
have a standard...

Rainer

> From: tom.petch [mailto:[email-masked]] 
> Sent: Friday, January 05, 2007 6:47 PM
> To: Rainer Gerhards; Glenn M. Keeni; [email-masked]
> Subject: Re: [Syslog] Syslog Protocol doubts
> 
> <inline>
> Tom Petch
>

---

From: "Glenn M. Keeni" <[email-masked]>
Date: Sat, 06 Jan 2007 13:22:44 +0900

Rainer,
> precisely describes the list consensus.
I will not have problems with that. The document needs to have
some statement like the above for completeness.
As for the MIB I will reword the two statistics like

    syslogEntityOperationsMsgsMalFormed
    DESCRIPTION:
         "The number of messages received by the syslog
          receiver which had malformed headers."

    syslogEntityOperationsMsgsDiscarded
    DESCRIPTION:
         "The number of messages that were discarded by the
          syslog receiver. This will include messages that
          were discarded because the message size was greater
          than the system's maximum message size."

That still leaves one question unanswered:

    Sec. 6.3  A receiver MAY ignore malformed STRUCTURED-DATA
              elements.
    Doubt:    But in sec. 6.4 Example 4 of a malformed
              STRUCTURED-DATA element the document says ...
              "the receiver MAY discard this message".

    Is the malformed STRUCTURED-DATA element ignored or the
    message as a whole is ignored?

Finally, a comment:
> The
> reason was always that MUSTs here are not actually needed
> to ensure interoperability.
if the MUST specifications are not a must for interoperability
then don't the MUSTs sound like an overkill ? Would "SHOULD"
be more appropriate? I understand that it is too late to ask
this question, but the quote above puts the MUSTs in a new
perspective. If this has been discussed and agreed upon, then
I have missed the discussion and I will withdraw the comment.

Glenn

---

From: "Rainer Gerhards" <[email-masked]>
Date: Sat, 6 Jan 2007 11:48:41 +0100

Hi Glenn, 

> Sent: Saturday, January 06, 2007 5:23 AM
> To: [email-masked]

I agree. I will log this for after IESG last call. The document is
submitted to the IESG, so I currently do not have change control over
it.

>           than the system's maximum message size."

I agree.

>     message as a whole is ignored?

I do not find these statements contradictionary. The receiver MAY ignore
the structure data element or it MAY discard the message. It is up to
the receiver. The wording was explicit to caution against sending it
malformed.

> I have missed the discussion and I will withdraw the comment.

I think it has been discussed ... Anyhow: have you read my response to
Tom Petch? I think it includes the reasoning. If you find my argument
insuffcient, please let me know. 

Rainer

---

From: "tom.petch" <[email-masked]>
Date: Sat, 6 Jan 2007 20:03:59 +0100

<inline>
Tom Petch

----- Original Message -----
From: "Rainer Gerhards" <[email-masked]>
To: "tom.petch" <[email-masked]>; "Glenn M. Keeni" <[email-masked]>;
<[email-masked]>
Sent: Friday, January 05, 2007 9:12 PM
Subject: RE: [Syslog] Syslog Protocol doubts

Hi Tom,

Rainer

<tp>
I disagree:-)

I am not suggesting we add or change anything in -protocol.  I think MUST is
well defined in RFC2119 and that the consequences are clear.  If the sender
demonstrably does not conform to -protocol in some regard (to a MUST), then what
confidence can you have that they will conform in other regards where that may
not be immediately apparent from the message itself?

syslog may have more MUSTs than most but I think that what makes syslog unusual
is that it is simplex; in other protocols, the receiver can say 'what on earth
is this?'; here it cannot and so I am content that the specification should be
stricter.

Tom Petch

---

From: "David Harrington" <[email-masked]>
Date: Mon, 8 Jan 2007 17:42:22 -0500

Hi,

[speaking as co-chair]

Let's make this more concrete. What specific changes to which specific
documents are you requesting?

David Harrington
[email-masked] 
[email-masked]
[email-masked]

> Sent: Saturday, January 06, 2007 2:04 PM

---

From: "tom.petch" <[email-masked]>
Date: Tue, 9 Jan 2007 09:38:37 +0100

----- Original Message -----
From: "David Harrington" <[email-masked]>
To: "'tom.petch'" <[email-masked]>; "'Rainer Gerhards'"
<[email-masked]>; "'Glenn M. Keeni'" <[email-masked]>;
<[email-masked]>
Sent: Monday, January 08, 2007 11:42 PM
Subject: RE: [Syslog] Syslog Protocol doubts
> documents are you requesting?

I am not proposing any change to -protocol -tls or -sign.

The tangential issue I did raise in response to your question about mib
structure was about having finer granularity in the counting of malformed
messages and I think Glenn's message points out the problem beautifully.
Suppose mib stats show 9000 messages arrived and 53 were malformed (and so were
discarded).  How do I
begin to look for them, when, from whom, what kind of malformation?  I wouldn't
even being looking but I might be worried.  Separate counters for types of
malformation, time of last malformed message (or limiting the scope of
LastError/LastErrorTime) would help.

Tom Petch