Subject: Separating the syslog Protocol from syslog-sign
From: Chris Lonvick <[email-masked]>
Date: Mon, 24 Nov 2003 08:06:20 -0800 (PST)

Hi Folks,

I've had some discussions with several people and have looked closely at
what we're doing with the definitions of the syslog protocol in the
syslog-sign ID along with what Rainer has been proposing for the
syslog-international ID.
http://www.ietf.org/internet-drafts/draft-ietf-syslog-sign-13.txt
http://www.ietf.org/internet-drafts/draft-ietf-syslog-international-00.txt

It appears that the idea of the cookies may be used for several future
items.  It doesn't seen appropriate to standardize the cookies in
syslog-sign so we've taken a look at producing separate documents for
syslog-sign and for the definition of the syslog-protocol.

The important part of the Working Group process is that we have consensus
on what we are doing and how we are getting there.  I'd like to ask Rainer
to produce syslog-protocol-00 with the concepts and protocol definitions
that we've been putting into the syslog-sign ID.  If that opens enough
doors for the future extensibility of the protocol then I feel that we can
quickly remove those parts from the current syslog-sign ID to produce a
mechanism that fulfils the intent of authenticating syslog messages and
yet remains transport independent.  I believe that we have consensus on
the signing aspects of syslog-sign at this time and that trimmed ID could
go to Last Call quickly so we can focus our attention on the syslog
protocol.  After all of the discussions we've had I suspect that the
syslog-protocol ID can also move on to Last Call quickly.

Is there anyone opposed to looking at a syslog-protocol-00 ID at this
time?  As said, I think that the effort to produce this should be minimal
since we've had so many good discussions about it in the past, and since
syslog-sign is in such good shape.  When that is entered into the ID
repository, I'd like to ask that we look at separating the protocol from
the contents of syslog-sign.  Again, that effort should be minimal and we
can progress the ID quickly from there as a mechanism that is independent
of the underlying syslog protocol but will add immense value where used.

Thanks,
Chris

------------------------------

Date: Wed, 03 Dec 2003 15:29:30 -0500
From: [email-masked]
Subject: I-D ACTION:draft-ietf-syslog-protocol-00.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Security Issues in Network Event Logging Working Group of the IETF.

	Title		: The syslog Protocol
	Author(s)	: R. Gerhards
	Filename	: draft-ietf-syslog-protocol-00.txt
	Pages		: 32
	Date		: 2003-12-3

This document describes the syslog protocol. The syslog protocol has
been used throughout the years to convey event notifications. This
documents describes a layered architecture for a backwards-compatible
and easily extensible syslog protocol.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-00.txt

To remove yourself from the IETF Announcement list, send a message to
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-syslog-protocol-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-syslog-protocol-00.txt".

NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

------------------------------

Date: Wed, 3 Dec 2003 18:40:24 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: New ID: The syslog Protocol

Hi Folks,

As is was announced, we have a new Working Group ID.  It may be found
here:

  http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-00.txt

I've updated our web page with this information as well.
  http://www.ietf.org/html.charters/syslog-charter.html

Does this look like a good separation of the syslog protocol from the
syslog-sign mechanism?  If so, then we can pull out the "protocol"
sections from the syslog-sign ID and utilize Rainer's syslog protocol ID
as the basis for syslog-international, and for any other additional
features that people may want in event logging.  (Those things may be
outside the scope of this WG but I'll allow discussion on the mailing
list.)I would like comments on this ID please.

I believe that this is identical to what we've currently specified in
syslog-sign.  Rainer has some additional thoughts that he would like to
discuss on the list.  If everyone could give this some close attention, we
can get this moved along pretty quickly.

Thanks,
Chris

------------------------------

Date: Mon, 15 Dec 2003 18:22:59 +0100
From: Rainer Gerhards <[email-masked]>
Subject: Draft Presentation on syslog-protocol

Hi all,

Chris has uploaded a draft version of my presentation on
syslog-protocol.

  http://www.employees.org/~lonvick/attachments/syslog-protocol.sxi
  http://www.employees.org/~lonvick/attachments/syslog-protocol.pdf

I first thought it needed more polishing before posting, but after some
more thinking I came to the conclusion that this polishing can only be
done after some issues have been discussed. So there it is.

The goal of the presentation is to show

- - the rationale behind creating syslog-protocol,
- - how it fits into the syslog family,
- - which open questions are there as of now

The target audience of this presentation is not strictly the WG, only. I
think once things are fully discussed, this (or additional) presentation
can also be used to educate implementors on what we actually want to
achive. From what I have seen in the past, finding acceptance for a
supposed standard is at least as important as doing it (technically)
right. What does it help if nobody implements it...?

So I think it is a good idea to develop such material in parallel to the
actual RFC. The plus is that at least I find such a presentation an easy
entry point into what the RFC tries to say.

I will post separate mails to some of the issues raised in the
presentation. You do NOT necessarily need to read the presentation in
order to understand my upcoming questions. I just think it will help
general understanding.

Cheers,
Rainer

------------------------------

Date: Mon, 15 Dec 2003 18:36:01 +0100
From: Rainer Gerhards <[email-masked]>
Subject: TAG incompatibility

Hi all,

I would like to drag your attention to this yet undiscussed posting:

http://www.mail-archive.com/syslog-sec%40employees.org/msg01338.html

It is regarding the syslog TAG. The format described was discussed in
the context of syslog-sign-13 but now also applies to
syslog-international-00.

There is an unresolved compatibility issue regarding the use of COLON in
the tag.

This is a quick excerpt of the issue:

####
So I think we need to make a tradeoff decision: we can either

a) allow colons in the path name
xor ;)
b) allow TAG NOT to be terminated by SP

Selecting
a) will break compliance for older clients (how many?)
b) will open up a can of (security?) bugs as I guess there are
   well enough implementors out there not caring about the
   restricted char
####

I would appreciate any feedback to this topic.

Many thanks,
Rainer

------------------------------

Date: Wed, 17 Dec 2003 18:55:06 +0100
From: Rainer Gerhards <[email-masked]>
Subject: syslog-protocol: Cookie Format

Hi WG,

Chris has proposed a XLM-like cookie format in private mail. I received
permission to quote him on this. I would appreciate any feedback from
the WG.

Chris said (Chris, please correct me if you feel I am quoting out of
context):

####
The thought just struck me about the use of cookies, languages, etc.
The
use of the special characters is what John Kelsey proposed.  We're not
limited to that if you can come up with something better.  I was
thinking
of something like the following at the start of the MSG:

    <cookie MSGNO=123 ENCODING=USASCII /> msg.msg.msg
or
    <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn /> msg.msg.msg
or even
    <cookie block=certblock /> certificate.block.msg

syslog-protocol would then define the IANA held cookie parameters and
vendors would be able to add their own.  Experimental parameters could
be
done like "vendorparam" (above) where the "%" replaces the "=" of a real
parameter.
####

Please note: In this quote, "msg.msg.msg" is a syslog message. It is NOT
(necessarily) multiple messages (it may for fragmented messages, but
that is a separate issue).

A key point in Chris suggestion is that this is NOT actual XML. It just
looks so. But we could specify the exact sequence of parameters and
such, so that no XML parser is needed to process it. After some initial
scepticism, I have to say that I like this approach. It looks very clean
and extensible.

In this mail, I am trying to gather some feedback from the WG if we
should move into the proposed direction. If so, I will focus on some of
the details.

Questions so: Do you like this? Do you think it is useful? Should we
proceed into this direction?

Rainer

------------------------------

Date: Wed, 17 Dec 2003 18:08:31 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Rainer:

Sorry if I did not follow earlier discussion and misunderstand
something obvious. But here is my feedback.

I like the key=value approach. A lot! This would be hugely welcome in
syslog and this is a great undertaking! To begin with, we could plug
things like severity and facility because, otherwise, they don't even
get stored for standard UDP syslog messages! I have dozens of
practical examples of other things that could be put into these tags.

I am not clear why it has to be enclosed specifically in XML-like tags
though. See what we did below for an alternative.

If "/>" is a special sequence, are there any rules for escaping it if
one needs to put it in tag values?

I am also not clear on the use of word "cookie" in general. Can you
explain where does this name concept come from?  The only association
that comes to my mind is with web browser cookies, and I don't see the
connection.  Can we call these key-value pairs "tags" or something
else?

At Cisco, we are evangelizing a new logging specification for logging
to file or through syslog. It is called CiscoLog.  It specifies
optional use of tags. We are looking at delimiting individual tags
with [] and the tag section with ": ".  Here is an example of a
message with tags as seen on the wire, but without the syslog PRI
field:

12: host.cisco.com: *Jun 13 2003 23:11:52.454 UTC:
%BACC-4-BAD_REQUEST: RDU:
[comp=parser][mac=1,6,aa:bb:cc:11:22:33][txn=mytxn123]: Bad request
received from device [1,6,aa:bb:cc:11:22:33]. Header missing.

Things in square brackets after "RDU: " are tags.  We don't follow
some of the optional syslog RFC3164 time and other header formats
partly for legacy reasons and also largely due to the fact that syslog
server implementations vary in what they store (or how they forward)
messages while we need a minimum set of data to always be present in
the log file and in a consistent format. I can give more details about
our format if needed. I am allowed to disclose it.

It is also useful to differentiate tag syntax from tag meaning.  We
addressed this with the use of "semantic extensions".  For example,
tag with key "ip" just means that the value is a properly formatted IP
related to message.  Tag "ip.orig" means it is IP of host originating
message.  Tag "ip.client" means it is a tag from a client talking to
server which originated the message.  You can also see things like
"ip.from", "ip.to".  Would it be useful to make the syntax/semantic
distinction in the standard?

BTW, we also use tags to mark parts of a multi-part messages.  So,
there is a lot of uses for them.

>     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn
> /> msg.msg.msg

Are spaces allowed for values?  They should be, I think.

Why a different syntax for experimental tags?

Getting back to syslog-international, I think tag values should also
be allowed to be in foreign languages eventually. Preferably UTF-8.
Tags could be used to provide any easily parseable parameters. If I
use "username" tag, for example, then in Japan the value could be in
Japanese.  We had some specific requirements from some Cisco apps to
support this. We support foreign languages in tag values and
specifying UTF-8 encoding for entire message (US-ASCII is preserved as
is).

Thanks,
Anton.

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Rainer Gerhards
> Sent: Wednesday, December 17, 2003 12:55 PM
> To: [email-masked]
> Subject: syslog-protocol: Cookie Format
>
>
> Hi WG,
>
> Chris has proposed a XLM-like cookie format in private
> mail. I received
> permission to quote him on this. I would appreciate any
> feedback from
> the WG.
>
> Chris said (Chris, please correct me if you feel I am quoting out of
> context):
>
> ####
> The thought just struck me about the use of cookies, languages, etc.
> The
> use of the special characters is what John Kelsey proposed.
>  We're not
> limited to that if you can come up with something better.  I was
> thinking
> of something like the following at the start of the MSG:
>
>     <cookie MSGNO=123 ENCODING=USASCII /> msg.msg.msg
> or
>     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn
> /> msg.msg.msg
> or even
>     <cookie block=certblock /> certificate.block.msg
>
> syslog-protocol would then define the IANA held cookie
> parameters and
> vendors would be able to add their own.  Experimental
> parameters could
> be
> done like "vendorparam" (above) where the "%" replaces the
> "=" of a real
> parameter.
> ####
>
> Please note: In this quote, "msg.msg.msg" is a syslog
> message. It is NOT
> (necessarily) multiple messages (it may for fragmented messages, but
> that is a separate issue).
>
> A key point in Chris suggestion is that this is NOT actual
> XML. It just
> looks so. But we could specify the exact sequence of parameters and
> such, so that no XML parser is needed to process it. After
> some initial
> scepticism, I have to say that I like this approach. It
> looks very clean
> and extensible.
>
> In this mail, I am trying to gather some feedback from the WG if we
> should move into the proposed direction. If so, I will
> focus on some of
> the details.
>
> Questions so: Do you like this? Do you think it is useful? Should we
> proceed into this direction?
>
> Rainer
>
>

------------------------------

Date: Thu, 18 Dec 2003 08:43:51 +0100
From: Albert Mietus <[email-masked]>
Subject: Re: TAG incompatibility

Rainer Gerhards writes:

 > It is regarding the syslog TAG. The format described was discussed in
 > the context of syslog-sign-13 but now also applies to
 > syslog-international-00.

You will understand my vote ;-)

 > So I think we need to make a tradeoff decision: we can either
 > a) allow colons in the path name
 > xor ;)
 > b) allow TAG NOT to be terminated by SP

I vote A.

 > Selecting
 > a) will break compliance for older clients (how many?)

No it will NOT,

Syslog-rfc* is (now) step by step, faraway by beeing 100% complice to
old implementations e.g. "long timestamp, DNS hostname, etc. This bit
will not change this.

Futher, I guess most implementation do not use the collon. It just
text for them! (The use PRI and "programname" only)

See you

BTW
I have got a quick look to yout presenation. But need more time to
give feedback. (But no error found) Like the "layer idea".

- -- 
ALbert Mietus
	Send prive mail to:      [email-masked]
	Send business mail to:  [email-masked]
	Don't send spam mail!

------------------------------

Date: Thu, 18 Dec 2003 08:42:29 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi,

There is a real demand in the industry for leveraging existing standards
rather than creating new languages or special syntaxes. The demand is to
reduce the number of special languages an operator needs to learn, and
to make it possible to integrate the data from one application type with
others.

If you're going to use XML at all, then I recommend using real XML so
the data specified can be read by other applications that already use
XML, and integrated with their other data. I'm not sure how many
applications will want to integrate with syslog cookies, but if there is
useful information in the cookie, then another application might be able
to use that info in combination with other info to create additional
features.

The use of a special syntax that looks like but is not XML, fails to
achieve either of the demands.

I also suggest you learn from the mistakes of others. SNMP uses a
special "adapted subset" of ASN.1 and that has caused lots of problems
over the years.

dbh
David Harrington
[email-masked]
co-chair, IETF SNMPv3 WG, concluded

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]]
> Sent: Wednesday, December 17, 2003 12:55 PM
> To: [email-masked]
> Subject: syslog-protocol: Cookie Format
>
>
> Hi WG,
>
> Chris has proposed a XLM-like cookie format in private mail.
> I received
> permission to quote him on this. I would appreciate any feedback from
> the WG.
>
> Chris said (Chris, please correct me if you feel I am quoting out of
> context):
>
> ####
> The thought just struck me about the use of cookies, languages, etc.
> The
> use of the special characters is what John Kelsey proposed.  We're not
> limited to that if you can come up with something better.  I was
> thinking
> of something like the following at the start of the MSG:
>
>     <cookie MSGNO=123 ENCODING=USASCII /> msg.msg.msg
> or
>     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn />
> msg.msg.msg
> or even
>     <cookie block=certblock /> certificate.block.msg
>
> syslog-protocol would then define the IANA held cookie parameters and
> vendors would be able to add their own.  Experimental parameters could
> be
> done like "vendorparam" (above) where the "%" replaces the
> "=" of a real
> parameter.
> ####
>
> Please note: In this quote, "msg.msg.msg" is a syslog
> message. It is NOT
> (necessarily) multiple messages (it may for fragmented messages, but
> that is a separate issue).
>
> A key point in Chris suggestion is that this is NOT actual
> XML. It just
> looks so. But we could specify the exact sequence of parameters and
> such, so that no XML parser is needed to process it. After
> some initial
> scepticism, I have to say that I like this approach. It looks
> very clean
> and extensible.
>
> In this mail, I am trying to gather some feedback from the WG if we
> should move into the proposed direction. If so, I will focus
> on some of
> the details.
>
> Questions so: Do you like this? Do you think it is useful? Should we
> proceed into this direction?
>
> Rainer
>
>
>

------------------------------

Date: Thu, 18 Dec 2003 06:13:51 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi Anton,

The term "cookies" was first used by John Kelsey in his first drafts of
syslog-sign to denote that the content of the message was somehow special
and should be parsed differently.  This sounded like a good idea then and
Rainer used the same approach when he started thinking about
syslog-international.  Since the cookies (or "tags" or "identifiers" or
whatever you wish to call them) still seem to be a good thing, I asked
Rainer to formalize them in syslog-protocol.

I did make the suggestion that it "look" like XML.  I'm very glad to get
the discussion going and will support the consensus of the WG so please,
everyone, comment on this issue.

Thanks,
Chris

On Wed, 17 Dec 2003, Anton Okmianski wrote:

> Rainer:
>
> Sorry if I did not follow earlier discussion and misunderstand
> something obvious. But here is my feedback.
>
> I like the key=value approach. A lot! This would be hugely welcome in
> syslog and this is a great undertaking! To begin with, we could plug
> things like severity and facility because, otherwise, they don't even
> get stored for standard UDP syslog messages! I have dozens of
> practical examples of other things that could be put into these tags.
>
> I am not clear why it has to be enclosed specifically in XML-like tags
> though. See what we did below for an alternative.
>
> If "/>" is a special sequence, are there any rules for escaping it if
> one needs to put it in tag values?
>
> I am also not clear on the use of word "cookie" in general. Can you
> explain where does this name concept come from?  The only association
> that comes to my mind is with web browser cookies, and I don't see the
> connection.  Can we call these key-value pairs "tags" or something
> else?
>
> At Cisco, we are evangelizing a new logging specification for logging
> to file or through syslog. It is called CiscoLog.  It specifies
> optional use of tags. We are looking at delimiting individual tags
> with [] and the tag section with ": ".  Here is an example of a
> message with tags as seen on the wire, but without the syslog PRI
> field:
>
> 12: host.cisco.com: *Jun 13 2003 23:11:52.454 UTC:
> %BACC-4-BAD_REQUEST: RDU:
> [comp=parser][mac=1,6,aa:bb:cc:11:22:33][txn=mytxn123]: Bad request
> received from device [1,6,aa:bb:cc:11:22:33]. Header missing.
>
> Things in square brackets after "RDU: " are tags.  We don't follow
> some of the optional syslog RFC3164 time and other header formats
> partly for legacy reasons and also largely due to the fact that syslog
> server implementations vary in what they store (or how they forward)
> messages while we need a minimum set of data to always be present in
> the log file and in a consistent format. I can give more details about
> our format if needed. I am allowed to disclose it.
>
> It is also useful to differentiate tag syntax from tag meaning.  We
> addressed this with the use of "semantic extensions".  For example,
> tag with key "ip" just means that the value is a properly formatted IP
> related to message.  Tag "ip.orig" means it is IP of host originating
> message.  Tag "ip.client" means it is a tag from a client talking to
> server which originated the message.  You can also see things like
> "ip.from", "ip.to".  Would it be useful to make the syntax/semantic
> distinction in the standard?
>
> BTW, we also use tags to mark parts of a multi-part messages.  So,
> there is a lot of uses for them.
>
> >     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn
> > /> msg.msg.msg
>
> Are spaces allowed for values?  They should be, I think.
>
> Why a different syntax for experimental tags?
>
> Getting back to syslog-international, I think tag values should also
> be allowed to be in foreign languages eventually. Preferably UTF-8.
> Tags could be used to provide any easily parseable parameters. If I
> use "username" tag, for example, then in Japan the value could be in
> Japanese.  We had some specific requirements from some Cisco apps to
> support this. We support foreign languages in tag values and
> specifying UTF-8 encoding for entire message (US-ASCII is preserved as
> is).
>
> Thanks,
> Anton.
>
>
>
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]]On Behalf Of Rainer Gerhards
> > Sent: Wednesday, December 17, 2003 12:55 PM
> > To: [email-masked]
> > Subject: syslog-protocol: Cookie Format
> >
> >
> > Hi WG,
> >
> > Chris has proposed a XLM-like cookie format in private
> > mail. I received
> > permission to quote him on this. I would appreciate any
> > feedback from
> > the WG.
> >
> > Chris said (Chris, please correct me if you feel I am quoting out of
> > context):
> >
> > ####
> > The thought just struck me about the use of cookies, languages, etc.
> > The
> > use of the special characters is what John Kelsey proposed.
> >  We're not
> > limited to that if you can come up with something better.  I was
> > thinking
> > of something like the following at the start of the MSG:
> >
> >     <cookie MSGNO=123 ENCODING=USASCII /> msg.msg.msg
> > or
> >     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn
> > /> msg.msg.msg
> > or even
> >     <cookie block=certblock /> certificate.block.msg
> >
> > syslog-protocol would then define the IANA held cookie
> > parameters and
> > vendors would be able to add their own.  Experimental
> > parameters could
> > be
> > done like "vendorparam" (above) where the "%" replaces the
> > "=" of a real
> > parameter.
> > ####
> >
> > Please note: In this quote, "msg.msg.msg" is a syslog
> > message. It is NOT
> > (necessarily) multiple messages (it may for fragmented messages, but
> > that is a separate issue).
> >
> > A key point in Chris suggestion is that this is NOT actual
> > XML. It just
> > looks so. But we could specify the exact sequence of parameters and
> > such, so that no XML parser is needed to process it. After
> > some initial
> > scepticism, I have to say that I like this approach. It
> > looks very clean
> > and extensible.
> >
> > In this mail, I am trying to gather some feedback from the WG if we
> > should move into the proposed direction. If so, I will
> > focus on some of
> > the details.
> >
> > Questions so: Do you like this? Do you think it is useful? Should we
> > proceed into this direction?
> >
> > Rainer
> >
> >
>
>
>

------------------------------

Date: Thu, 18 Dec 2003 15:01:59 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi David and others,

> There is a real demand in the industry for leveraging
> existing standards
> rather than creating new languages or special syntaxes. The
> demand is to
> reduce the number of special languages an operator needs to learn, and
> to make it possible to integrate the data from one
> application type with
> others.

I am glad to hear that the overall opinion is that the xml based syntax
seems to be a good idea.

> If you're going to use XML at all, then I recommend using real XML so
> the data specified can be read by other applications that already use
> XML, and integrated with their other data. I'm not sure how many
> applications will want to integrate with syslog cookies, but
> if there is
> useful information in the cookie, then another application
> might be able
> to use that info in combination with other info to create additional
> features.

Actually I am looking at the same subset that is used in BEEP (RFC3080).
The goal is that we do not allow constructs that it would be hard to
find a good reasoning for (in syslog).

Even with RFC 3080, which is used as transport for RFC3195, there was a
looot of discussion both inside and outside of this WG from syslog
developers who said "this is far too complex, we will not implement it".
As far as I can see, this very same argument is still around and one of
the reasons why there are few implementations of 3195 (I have to say,
though, that I initially had the same opinion and prooved it to be false
when I implemented 3195 - so it is maybe just an education issue...).

So my effort is to satisfy the "simple enough" and "I will not need to
include an XML parser in my app" arguments while still being able to
provide a more standardized form of the cookie format. As some pointed
out, this XML based approach could - in the long term - also be used to
structure syslog payload, which is currently beyond the charter of this
WG.

I currently can not see any harm by insisting on "simple XML" for
cookies. But, yes, you are right, the more one thinks about yet
unexplored options (payload structure!) the more it can become a bad
idea.

The pro side of simple XML is that it is indeed very simple to use and
has kind of the "syslog spirit for simplicity".

> The use of a special syntax that looks like but is not XML, fails to
> achieve either of the demands.

To an XML parser, it *is* XML when reading. To an app without an
XML-parser, it is textual data that you can parse with a simple,
custom-build parser.

The issue is when writing. There, an XML parser may generate more
elaborate format, which may break the custom-build parser.

Do you think that RFC3080 defined BEEP XML *is* a compromise between
no-XML and full-XML?

> I also suggest you learn from the mistakes of others. SNMP uses a
> special "adapted subset" of ASN.1 and that has caused lots of problems
> over the years.

I am listening very closely. My response is to describe what is behind
my XML format suggestion. Which goals I (we?) try to tackle.

Did the SNMP ASN.1 subset decision have similar reasons? If so, is this
a good indication the the very same reasoning will not work for syslog,
too? Is it a comparable case?

Thanks for taking the time to look into this issue. I appreciate your
feedback very much!

Rainer

------------------------------

Date: Thu, 18 Dec 2003 11:09:07 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi Rainer,

Let me start about by saying that I am often curmudgeonly, so please
forgive any negative tone to this message; I am making the comment to be
constructively critical. I have had experience dealing with problems
similar to those I find in this proposal, so please pay attention to the
concerns.

I want to clarify my position, which you aggregated with other
positions, and inadvertently misstated. Since it has my name in the
greeting, I would like to make sure your misstatement doesn't mislead
anybody about my position.

I did not say that XML-based syntax is a good idea. It may be; it may
not be; whether it is will depend on lots of factors that need to be
weighed carefully. Two of the potential benefits of using XML are reuse
of existing implemented standards (which benefits both operators and
implementors) and interoprability with other application formats (which
benefits application developers and consequently application users).

I think an XML-"like" syntax may be a BAD idea because it achieves
neither of these two benefits, and adds the negative tradeoffs of having
to maintain the new XML-like syntax, requiring operators to learn a
unique syntax for syslog, and having to support a unique new syntax in
implementations. Making it "look like" XML doesn't mean it gets the
benefits of using XML.

Beyond this point, I make only one other comment about the proposal.
Using existing terms for existing standards to refer to non-compliant
"look-like" things benefits few. It is misleading and adds confusion to
the already difficult process of standardization. I think it is a bad
idea to refer to your proposal as "simple XML" - it is either XML or it
is not - or to refer to the information as a "cookie", which has a
commonly accepted usage which this usage is not.

My $.02 (or $.01).

Thanks,
dbh

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]]
> Sent: Thursday, December 18, 2003 9:02 AM
> To: Harrington, David; [email-masked]
> Subject: RE: syslog-protocol: Cookie Format
>
>
> Hi David and others,
>
> > There is a real demand in the industry for leveraging
> > existing standards
> > rather than creating new languages or special syntaxes. The
> > demand is to
> > reduce the number of special languages an operator needs to
> learn, and
> > to make it possible to integrate the data from one
> > application type with
> > others.
>
> I am glad to hear that the overall opinion is that the xml
> based syntax
> seems to be a good idea.
>
> > If you're going to use XML at all, then I recommend using
> real XML so
> > the data specified can be read by other applications that
> already use
> > XML, and integrated with their other data. I'm not sure how many
> > applications will want to integrate with syslog cookies, but
> > if there is
> > useful information in the cookie, then another application
> > might be able
> > to use that info in combination with other info to create additional
> > features.
>
> Actually I am looking at the same subset that is used in BEEP
> (RFC3080).
> The goal is that we do not allow constructs that it would be hard to
> find a good reasoning for (in syslog).
>
> Even with RFC 3080, which is used as transport for RFC3195,
> there was a
> looot of discussion both inside and outside of this WG from syslog
> developers who said "this is far too complex, we will not
> implement it".
> As far as I can see, this very same argument is still around
> and one of
> the reasons why there are few implementations of 3195 (I have to say,
> though, that I initially had the same opinion and prooved it
> to be false
> when I implemented 3195 - so it is maybe just an education issue...).
>
> So my effort is to satisfy the "simple enough" and "I will not need to
> include an XML parser in my app" arguments while still being able to
> provide a more standardized form of the cookie format. As some pointed
> out, this XML based approach could - in the long term - also
> be used to
> structure syslog payload, which is currently beyond the
> charter of this
> WG.
>
> I currently can not see any harm by insisting on "simple XML" for
> cookies. But, yes, you are right, the more one thinks about yet
> unexplored options (payload structure!) the more it can become a bad
> idea.
>
> The pro side of simple XML is that it is indeed very simple to use and
> has kind of the "syslog spirit for simplicity".
>
> > The use of a special syntax that looks like but is not XML, fails to
> > achieve either of the demands.
>
> To an XML parser, it *is* XML when reading. To an app without an
> XML-parser, it is textual data that you can parse with a simple,
> custom-build parser.
>
> The issue is when writing. There, an XML parser may generate more
> elaborate format, which may break the custom-build parser.
>
> Do you think that RFC3080 defined BEEP XML *is* a compromise between
> no-XML and full-XML?
>
> > I also suggest you learn from the mistakes of others. SNMP uses a
> > special "adapted subset" of ASN.1 and that has caused lots
> of problems
> > over the years.
>
> I am listening very closely. My response is to describe what is behind
> my XML format suggestion. Which goals I (we?) try to tackle.
>
> Did the SNMP ASN.1 subset decision have similar reasons? If
> so, is this
> a good indication the the very same reasoning will not work
> for syslog,
> too? Is it a comparable case?
>
> Thanks for taking the time to look into this issue. I appreciate your
> feedback very much!
>
> Rainer
>

------------------------------

Date: Thu, 18 Dec 2003 17:38:33 +0100
From: Rainer Gerhards <[email-masked]>
Subject: Name for "cookies"

Hi WG,

as David Harrington correctly pointed out, the name "cookie" may be
misleading for what we are currently using it. I now suggest that we use
an alternate name, just to avoid confusion.

In my view, this is a kind of meta data. So I propose to call it
"metadata" or just "meta". I am not sure if this, too can be misleading
(meta tags in html), but I have to admit I do not have a better idea.

Any comments or suggestions?

Rainer

------------------------------

Date: Thu, 18 Dec 2003 17:27:41 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi David,

> Let me start about by saying that I am often curmudgeonly, so please
> forgive any negative tone to this message; I am making the
> comment to be
> constructively critical. I have had experience dealing with problems
> similar to those I find in this proposal, so please pay
> attention to the
> concerns.

I have no problem with that (besides that I don't find a negative
tone;)). I value your opinion much as you obviously have experience in
this regard.

> I want to clarify my position, which you aggregated with other
> positions, and inadvertently misstated.

sorry - that actually was what I understood...

> Since it has my name in the
> greeting, I would like to make sure your misstatement doesn't mislead
> anybody about my position.
>
> I did not say that XML-based syntax is a good idea. It may be; it may
> not be; whether it is will depend on lots of factors that need to be
> weighed carefully. Two of the potential benefits of using XML
> are reuse
> of existing implemented standards (which benefits both operators and
> implementors) and interoprability with other application
> formats (which
> benefits application developers and consequently application users).
>
> I think an XML-"like" syntax may be a BAD idea because it achieves
> neither of these two benefits, and adds the negative
> tradeoffs of having
> to maintain the new XML-like syntax, requiring operators to learn a
> unique syntax for syslog, and having to support a unique new syntax in
> implementations. Making it "look like" XML doesn't mean it gets the
> benefits of using XML.

I see the argument. Do you think that relying on something already
accepted - like BEEP XML in RFC3080 is a solution? As far as I can see,
RFC3080 is increasingly becoming implemented. So while not full XML, it
may be a standard that works "as expected"?

>
> Beyond this point, I make only one other comment about the proposal.
> Using existing terms for existing standards to refer to non-compliant
> "look-like" things benefits few. It is misleading and adds
> confusion to
> the already difficult process of standardization.

... understood and agreed.

> I think it is a bad
> idea to refer to your proposal as "simple XML" - it is either
> XML or it
> is not - or to refer to the information as a "cookie", which has a
> commonly accepted usage which this usage is not.

This, too is a very valid point. I never thought about it, just picked
the word from syslog-sign.

I think we (syslog-sec WG) should probably think about another term. I
will spawn off a new thread for this.

Thanks again for your very helpful comments!

Rainer

------------------------------

Date: Thu, 18 Dec 2003 12:53:32 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi Rainer,

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]]
> I see the argument. Do you think that relying on something already
> accepted - like BEEP XML in RFC3080 is a solution? As far as
> I can see,
> RFC3080 is increasingly becoming implemented. So while not
> full XML, it
> may be a standard that works "as expected"?
>

OK, you're dragging me kicking and screaming into this discussion. ;-)

BEEP ... Difficult problem. Is this existing technology to leverage or
not?

As a CTO architect for a major equipment vendor, I have concerns about
whether BEEP is being accepted by the industry. I am not yet
recommending that my company implement it in our devices because I am
not seeing customer demand for it, competitive pressures for it,
engineers' excitement about this new technology, or adoption by other
standards bodies. I am not yet seeing it implemented in the devices of
major equipment vendors. I suspect that means other companies are also
not seeing customer demand for BEEP. Since BEEP is really an engineering
infrastructural element rather than a customer feature, that's to be
somewhat expected, but I suspect other companies are also not seeing
engineering demand for BEEP.

Among IETF protocol designers, there is fairly strong promotion and
acceptance of BEEP, and a number of WGs are building upon it.
Personally, as a protocol designer myself, I think BEEP is lightweight
and serves valuable purposes, especially the ability to simplify
leveraging existing protocols.

A few WGs have chosen to build upon BEEP, but I haven't seen demand for
the protocols built upon BEEP either, and I suspect part of that is the
dependency on the BEEP protocol which isn't currently available in the
devices. Adopting these new protocols involves implementing not just one
unproven technology; it requires implementing two unproven technologies.
I haven't seen IDWG widely accepted in the IDS market; I haven't seen
vendors building the BEEP-dependent syslog proposals. NetConf is still
debating the same issue of whether BEEP has yet been adopted widely, and
is planning to ask the operators at NANOG next month whether they want
the features of BEEP in the Netconf solution.

http://www.ietf.org/IESG/implementation.html shows no implementation
reports for BEEP, IDWG, syslog-reliable, etc.

I've checked the beepcore and roadrunner BEEP implementation web sites,
and searched the web with Google, and I turn up a few open source
applications, but I don't see major software or hardware vendors
represented.

As the NETCONF work shows, the trend is moving toward acceptance of XML
and toward Web Services. BEEP may not go far enough to compete with, and
interoperate with, the likes of protocols supported in web services such
as SOAP. Part of the issue is, if the goal is leveraging existing
protocols and interoperability between protocols, and if web services
already promises these benefits, then does BEEP offer too little to
justify adding it to the protocols supported on devices, if we might
want to support more full-fledged protocols for web services? I think
the industry has not decided it wants to go all the way to web services
level integration yet, has decided it wants something better than a
free-for-all, but isn't convinced that moving to BEEP is the right
answer.

Whether a proposed standard really becomes a standard is far more
dependent on whether the industry accepts it than on what the IETF says.
I recommend the WG look closely to see whether BEEP is really being
accepted by the industry, or just by other WGs within the IETF, before
making a decision to build a dependency on BEEP or to emulate the
XML-subset design of BEEP.

I would love to see some solid information to show that major vendors
are implementing BEEP before I see this (or any other) WG create
dependencies on it.

dbh

------------------------------

Date: Thu, 18 Dec 2003 12:53:32 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hi Rainer,

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]]
> I see the argument. Do you think that relying on something already
> accepted - like BEEP XML in RFC3080 is a solution? As far as
> I can see,
> RFC3080 is increasingly becoming implemented. So while not
> full XML, it
> may be a standard that works "as expected"?
>

OK, you're dragging me kicking and screaming into this discussion. ;-)

BEEP ... Difficult problem. Is this existing technology to leverage or
not?

As a CTO architect for a major equipment vendor, I have concerns about
whether BEEP is being accepted by the industry. I am not yet
recommending that my company implement it in our devices because I am
not seeing customer demand for it, competitive pressures for it,
engineers' excitement about this new technology, or adoption by other
standards bodies. I am not yet seeing it implemented in the devices of
major equipment vendors. I suspect that means other companies are also
not seeing customer demand for BEEP. Since BEEP is really an engineering
infrastructural element rather than a customer feature, that's to be
somewhat expected, but I suspect other companies are also not seeing
engineering demand for BEEP.

Among IETF protocol designers, there is fairly strong promotion and
acceptance of BEEP, and a number of WGs are building upon it.
Personally, as a protocol designer myself, I think BEEP is lightweight
and serves valuable purposes, especially the ability to simplify
leveraging existing protocols.

A few WGs have chosen to build upon BEEP, but I haven't seen demand for
the protocols built upon BEEP either, and I suspect part of that is the
dependency on the BEEP protocol which isn't currently available in the
devices. Adopting these new protocols involves implementing not just one
unproven technology; it requires implementing two unproven technologies.
I haven't seen IDWG widely accepted in the IDS market; I haven't seen
vendors building the BEEP-dependent syslog proposals. NetConf is still
debating the same issue of whether BEEP has yet been adopted widely, and
is planning to ask the operators at NANOG next month whether they want
the features of BEEP in the Netconf solution.

http://www.ietf.org/IESG/implementation.html shows no implementation
reports for BEEP, IDWG, syslog-reliable, etc.

I've checked the beepcore and roadrunner BEEP implementation web sites,
and searched the web with Google, and I turn up a few open source
applications, but I don't see major software or hardware vendors
represented.

As the NETCONF work shows, the trend is moving toward acceptance of XML
and toward Web Services. BEEP may not go far enough to compete with, and
interoperate with, the likes of protocols supported in web services such
as SOAP. Part of the issue is, if the goal is leveraging existing
protocols and interoperability between protocols, and if web services
already promises these benefits, then does BEEP offer too little to
justify adding it to the protocols supported on devices, if we might
want to support more full-fledged protocols for web services? I think
the industry has not decided it wants to go all the way to web services
level integration yet, has decided it wants something better than a
free-for-all, but isn't convinced that moving to BEEP is the right
answer.

Whether a proposed standard really becomes a standard is far more
dependent on whether the industry accepts it than on what the IETF says.
I recommend the WG look closely to see whether BEEP is really being
accepted by the industry, or just by other WGs within the IETF, before
making a decision to build a dependency on BEEP or to emulate the
XML-subset design of BEEP.

I would love to see some solid information to show that major vendors
are implementing BEEP before I see this (or any other) WG create
dependencies on it.

dbh

------------------------------

Date: Thu, 18 Dec 2003 17:28:44 -0500
From: Doug Dunlop <[email-masked]>
Subject: Syslog message size, internationalization and IHE

Hi All,

I just wanted to bring into the discussion what is perhaps a new consideration for syslog message size and internationalization.

The Integrating the Healthcare Enterprise (IHE) initiative has specified the use of syslog as the mechanism for logging audit record messages. The specification allows up to 32Kb messages, the content of the MSG field is XML with a specified XML schema and it uses UTF-8 encodings.

The MSG content is intended to be easily machine readable, but not necessarily human readable. The major use case is to be able to determine which system users accessed which patient data, when they did it and what type of access they did.

This specification is given in the IHE Technical Framework, Basic Security Profile, Record Audit Event transaction.

IHE Home:
http://www.rsna.org/ihe

IHE Vol I, Integration Profiles:
http://www.rsna.org/IHE/tf/ihe_tf_rev5.5_vol1_TI.pdf

IHE Vol III, Transactions II (covers Auditing):
http://www.rsna.org/IHE/tf/ihe_tf_rev5.5_vol3_TI.pdf

There are 3 features needed in syslog to support IHE's Basic Security Profile.

1. Up to 32K message size
2. UTF-8 character encoding
3. XML tags in the MSG

It also makes sense that the TCP transport would be more reliable than UDP, even if not completely reliable. Only UDP transport is currently specified in IHE, but there are vague references to allowing for reliable syslog and BEEP in future versions of the specification.

Comments?

Regards,
Doug

- -----------------
Douglas Dunlop
AGFA HealthCare

------------------------------

Date: Fri, 19 Dec 2003 17:45:20 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: Syslog message size, internationalization and IHE

Hi Doug,

thanks for your interesting mail. I will reply in more detail when I am
through with the papers (looks like it takes some time), but I have an
immediate comment...

> The Integrating the Healthcare Enterprise (IHE) initiative
> has specified the use of syslog as the mechanism for logging
> audit record messages. The specification allows up to 32Kb
> messages, the content of the MSG field is XML with a
> specified XML schema and it uses UTF-8 encodings.

There is a good reason for the current 1k limit on syslog messages. This
1k is on most networks being transmitted in a single packet, which will
remove any fragmentation or multi-packet issues. With 32k, the data will
definitely travel in more than one packet (in almost all cases). So with
UDP, you are seeing challenges in in-order delivery and partial packet
loss.

I am trying to address this (at least partly) in the syslog-protocol
spec. In there, the idea is to basically stick with the 1k transport
limitation but - on an upper layer - support fragmentation and longer
messages with what we currently call a "cookie". This approach will
allow you to at least

a) detect missing packetc
b) reassmeble packets that were delivered out-of-order
   (with some burden on the reciver)

To be more precise, in b) we reassamble an upper-layer message that was
sent via multiple packets. That upper layer is what your application is
talking to.

More comments when I have read the other docs.

I would also like to use this message to whish all WG members a merry
xmas, a great new year and a peaceful holiday season. I hope it is
eventless from the infosec community point of view ;)

Rainer

------------------------------

Date: Fri, 19 Dec 2003 09:11:24 -0800
From: Darren New <[email-masked]>
Subject: Re: syslog-protocol: Cookie Format

Rainer Gerhards wrote:
> I see the argument. Do you think that relying on something already
> accepted - like BEEP XML in RFC3080 is a solution? As far as I can see,
> RFC3080 is increasingly becoming implemented. So while not full XML, it
> may be a standard that works "as expected"?

I believe that RFC3080 specifies the use of actual XML, not any
simplified version. It does insist on XML 1.0, and allows dropping of
the <? ...> headers at the start (since you're not passing an
independent document but rather an XML fragment inside another protocol,
and that protocol tells you that you're getting XML there). But in 2 of
the 3 implementations I've seen the innards of, it's using off-the-shelf
XML parsers. (And the reason the third one didn't is we didn't want to
make it dependent on outside libraries, allowing porting to (say)
routers or hand-held PDAs or whatever).

However, the examples I've seen such as
     <cookie MSGNO=123 ENCODING=USASCII />
are not valid XML tags, missing the quote marks around attribute values.

Rather than "simplified XML", consider "restricted XML", which is what I
think you're talking about wrt RFC3080. But this is a restriction on
allowable DTDs, not a simplification on the syntax.

- -- 
Darren New, San Diego CA USA (PST)
The standard of success is not perfection.
The standard of success is the alternative.

------------------------------

Date: Wed, 24 Dec 2003 12:51:34 -0500
From: Doug Dunlop <[email-masked]>
Subject: RE: Syslog message size, internationalization and IHE

Hi Rainer,

Thanks for your response, it was helpful, and I look forward to reading your additional comments. I just wanted to point out an additional spec that you may already know about, but I wanted to make sure.

This IETF draft is the proposed "new" IHE message format with XML schema.
http://www.ietf.org/internet-drafts/draft-marshall-security-audit-08.txt

Regards,
Doug

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]]
> Sent: Friday, December 19, 2003 11:45 AM
> To: Doug Dunlop; [email-masked]
> Subject: RE: Syslog message size, internationalization and IHE
>
>
> Hi Doug,
>
> thanks for your interesting mail. I will reply in more detail
> when I am
> through with the papers (looks like it takes some time), but I have an
> immediate comment...
>
> > The Integrating the Healthcare Enterprise (IHE) initiative
> > has specified the use of syslog as the mechanism for logging
> > audit record messages. The specification allows up to 32Kb
> > messages, the content of the MSG field is XML with a
> > specified XML schema and it uses UTF-8 encodings.
>
> There is a good reason for the current 1k limit on syslog
> messages. This
> 1k is on most networks being transmitted in a single packet,
> which will
> remove any fragmentation or multi-packet issues. With 32k,
> the data will
> definitely travel in more than one packet (in almost all
> cases). So with
> UDP, you are seeing challenges in in-order delivery and partial packet
> loss.
>
> I am trying to address this (at least partly) in the syslog-protocol
> spec. In there, the idea is to basically stick with the 1k transport
> limitation but - on an upper layer - support fragmentation and longer
> messages with what we currently call a "cookie". This approach will
> allow you to at least
>
> a) detect missing packetc
> b) reassmeble packets that were delivered out-of-order
>    (with some burden on the reciver)
>
> To be more precise, in b) we reassamble an upper-layer
> message that was
> sent via multiple packets. That upper layer is what your
> application is
> talking to.
>
> More comments when I have read the other docs.
>
> I would also like to use this message to whish all WG members a merry
> xmas, a great new year and a peaceful holiday season. I hope it is
> eventless from the infosec community point of view ;)
>
> Rainer
>
>
>

------------------------------

Date: Tue, 30 Dec 2003 17:39:57 +0100
From: Rainer Gerhards <[email-masked]>
Subject: commented archive

Hi WG,

I have created an (partial) archive of WG postings. Other than the full,
official one, this is manually structured according the contents of the
messages (at least as far as *I* think what content they belong to).

I had the idea to do this when I began to track the issues for
syslog-protocol. I have to admit I do not know if it is a good idea or
not. Also, I do not know if I can manage to keep it updated once the
message volume on the list increases. Let's see. If you find it useful
(or not useful;)) or have any other comments - please email my
privately. I would be very interested if you think this is a valuable
effort (most importantly because if all find it is totally useless, I
will probably spend the time somewhere else ;)).

The archive is at

    http://www.syslog.cc/ietf/

I would also like to wish all of you a happy and healthy new year!

Rainer

------------------------------