Subject: [PMOL] Re: A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)]
From: Yangwoo Ko <[email-masked]>
To: Leslie Daigle <[email-masked]>
Cc: IESG <[email-masked]>, [email-masked], [email-masked]
Date: Tue, 30 Oct 2007 12:06:45 +0900
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

Since application developers have developed many tricks to smooth out 
some peculiarities of network-based protocols (e.g. jitter, delay, drops 
and so on), it is very often hard to related the performance 
measured/perceived at application layer with those of underlying layers. 
If this WG provides guidelines (or at least hints) to get over this 
vagueness, I will be very happy.

Regards

Leslie Daigle wrote:
> 
> 
> At the Application Performance Metrics BoF in
> Chicago, there was a lot of discussion about whether
> or how to best capture expertise in the area of performance
> metrics for application to IETF protocols.
> 
> There seemed to be 2 separate questions on the table:
> 
>     1/ how, and
>     2/ whether
> 
> to generalize a performance metric framework for IETF
> protocols.
> 
> 
> The proposed PMOL WG addresses the first question -- how
> to do it.
> 
> On the second point -- the question is really about whether
> the IETF community as a whole supports/values performance
> metrics and will invest the effort in using any general
> framework for existing/new protocols.
> 
> I believe the folks that joined the PMOL mailing list after
> the APM BoF are assuming that support is there.
> 
> I suggest that now is an excellent time for folks _not_
> involved in the PMOL effort, but who are working on
> protocols that could/should make use of its output, to
> voice some opinions on whether or not this approach
> is helpful/will be useful to them in future protocols.
> 
> 
> Leslie.
> 
> 
> -------- Original Message --------
> Subject: WG Review: Performance Metrics at Other Layers (pmol)
> Date: Mon, 22 Oct 2007 14:15:02 -0400
> From: IESG Secretary <[email-masked]>
> Reply-To: [email-masked]
> To: [email-masked]
> 
> A new IETF working group has been proposed in the Operations and
> Management Area.  The IESG has not made any determination as yet.
> The following draft charter was submitted, and is provided for
> informational purposes only.  Please send your comments to the IESG
> mailing list ([email-masked]) by October 29.
> 
> +++
> 
> Performance Metrics at Other Layers (pmol)
> ==============================================
> 
> Current Status: Proposed Working Group
> 
> WG Chairs:
> TBD
> 
> Operations and Management Area:
> Dan Romascanu <[email-masked]>
> Ronald Bonica <[email-masked]>
> 
> Description:
> 
> The successful implementation and operation of IP based applications
> often depends on some underlying performance measurement infrastructure
> that helps service operators or network managers to recognize when
> performance is unsatisfactory and identify problems affecting service
> quality. Standardized performance metrics add the desirable features of
> consistent implementation, interpretation, no comparison.
> 
> The IETF has two Working Groups dedicated to the development of
> performance metrics however each has strict limitations in their
> charters:
> 
> - The Benchmarking Methodology WG has addressed a range of networking
> technologies and protocols in their long history (such as IEEE 802.3,
> ATM, Frame Relay, and Routing Protocols), but the charter strictly
> limits their Performance characterizations to the laboratory
> environment.
> 
> - The IP Performance Metrics WG has the mandate to develop metrics
> applicable to the performance of Internet data delivery, but it is
> specifically prohibited from developing metrics that characterize
> traffic (such as a VoIP stream).
> 
> The IETF also has current and completed activities related to the
> reporting of application performance metrics (e.g. RAQMON and RTCP XR)
> and is also actively involved in the development of reliable transport
> protocols which would affect the relationship between IP performance and
> application performance.
> 
> Thus there is a gap in the currently chartered coverage of IETF WGs:
> development of performance metrics for IP-based protocols and
> applications that operate over UDP, TCP, SCTP, DCCP, Forward Error
> Correction (FEC) and other robust transport protocols, and that can be
> used to characterize traffic on live networks.
> 
> The working group will focus on the completion of two RFCs:
> 
> 1. A PMOL framework and guidelines memo that will describe the necessary
> elements of performance metrics of protocols and applications
> transported over IETF-specified protocols (such as the formal
> definition, purpose, and units of measure) and the various types of
> metrics that characterize traffic on live networks (such as metrics
> derived from other metrics, possibly on lower layers). The framework
> will also address the need to specify the intended audience and the
> motivation for the performance metrics. There will also be guidelines
> for a performance metric development process that includes entry
> criteria for new proposals (how a proposal might be evaluated for
> possible endorsement by a protocol development working group), and how
> an successful proposal will be developed by PMOL WG in cooperation with
> a protocol development WG.
> 
> 2. A proof-of-concept RFC defining performance metrics for SIP, based on
> draft-malas-performance-metrics. This memo would serve as an example of
> the framework and the PMOL development process in the IETF.
> 
> Discussion of new work proposals is strongly discouraged under the
> initial charter of the PMOL WG, except to advise a protocol development
> WG when they are evaluating a new work proposal for related performance
> metrics.
> 
> The PMOL WG will also be guided by a document describing how memos
> defining performance metrics are intended to advance along the IETF
> Standards track (draft-bradner-metricstest).
> 
> PMOL WG will take advantage of expertise and seek to avoid overlap with
> other standards development organizations, such as ETSI STQ, ITU-T SG
> 12, ATIS IIF, ATIS PRQC, and others.
> 
> Milestones
> 
> June 08 SIP Performance Metrics Draft to IESG Review for consideration
> as Proposed Standard
> 
> Sept 08 PMOL Framework and Guidelines Draft to IESG Review for
> consideration as BCP
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ietf-announce
> 

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

From: Sam Hartman <[email-masked]>
To: Leslie Daigle <[email-masked]>
Date: Mon, 29 Oct 2007 16:24:02 -0400
Cc: IESG <[email-masked]>, [email-masked], [email-masked]
Subject: [PMOL] Re: A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)]

>>>>> "Leslie" == Leslie Daigle <[email-masked]> writes:

I doubt I'll use the output in security protocols.

    Leslie> Leslie.

    Leslie> -------- Original Message -------- Subject: WG Review:
    Leslie> Performance Metrics at Other Layers (pmol) Date: Mon, 22
    Leslie> Oct 2007 14:15:02 -0400 From: IESG Secretary
    Leslie> <[email-masked]> Reply-To: [email-masked] To:
    Leslie> [email-masked]

    Leslie> A new IETF working group has been proposed in the
    Leslie> Operations and Management Area.  The IESG has not made any
    Leslie> determination as yet.  The following draft charter was
    Leslie> submitted, and is provided for informational purposes
    Leslie> only.  Please send your comments to the IESG mailing list
    Leslie> ([email-masked]) by October 29.

    Leslie> +++

    Leslie> Performance Metrics at Other Layers (pmol)
    Leslie> ==============================================

    Leslie> Current Status: Proposed Working Group

    Leslie> WG Chairs: TBD

    Leslie> Operations and Management Area: Dan Romascanu
    Leslie> <[email-masked]> Ronald Bonica <[email-masked]>

    Leslie> Description:

    Leslie> The successful implementation and operation of IP based
    Leslie> applications often depends on some underlying performance
    Leslie> measurement infrastructure that helps service operators or
    Leslie> network managers to recognize when performance is
    Leslie> unsatisfactory and identify problems affecting service
    Leslie> quality. Standardized performance metrics add the
    Leslie> desirable features of consistent implementation,
    Leslie> interpretation, no comparison.

    Leslie> The IETF has two Working Groups dedicated to the
    Leslie> development of performance metrics however each has strict
    Leslie> limitations in their charters:

    Leslie> - The Benchmarking Methodology WG has addressed a range of
    Leslie> networking technologies and protocols in their long
    Leslie> history (such as IEEE 802.3, ATM, Frame Relay, and Routing
    Leslie> Protocols), but the charter strictly limits their
    Leslie> Performance characterizations to the laboratory
    Leslie> environment.

    Leslie> - The IP Performance Metrics WG has the mandate to develop
    Leslie> metrics applicable to the performance of Internet data
    Leslie> delivery, but it is specifically prohibited from
    Leslie> developing metrics that characterize traffic (such as a
    Leslie> VoIP stream).

    Leslie> The IETF also has current and completed activities related
    Leslie> to the reporting of application performance metrics
    Leslie> (e.g. RAQMON and RTCP XR) and is also actively involved in
    Leslie> the development of reliable transport protocols which
    Leslie> would affect the relationship between IP performance and
    Leslie> application performance.

    Leslie> Thus there is a gap in the currently chartered coverage of
    Leslie> IETF WGs: development of performance metrics for IP-based
    Leslie> protocols and applications that operate over UDP, TCP,
    Leslie> SCTP, DCCP, Forward Error Correction (FEC) and other
    Leslie> robust transport protocols, and that can be used to
    Leslie> characterize traffic on live networks.

    Leslie> The working group will focus on the completion of two
    Leslie> RFCs:

    Leslie> 1. A PMOL framework and guidelines memo that will describe
    Leslie> the necessary elements of performance metrics of protocols
    Leslie> and applications transported over IETF-specified protocols
    Leslie> (such as the formal definition, purpose, and units of
    Leslie> measure) and the various types of metrics that
    Leslie> characterize traffic on live networks (such as metrics
    Leslie> derived from other metrics, possibly on lower layers). The
    Leslie> framework will also address the need to specify the
    Leslie> intended audience and the motivation for the performance
    Leslie> metrics. There will also be guidelines for a performance
    Leslie> metric development process that includes entry criteria
    Leslie> for new proposals (how a proposal might be evaluated for
    Leslie> possible endorsement by a protocol development working
    Leslie> group), and how an successful proposal will be developed
    Leslie> by PMOL WG in cooperation with a protocol development WG.

    Leslie> 2. A proof-of-concept RFC defining performance metrics for
    Leslie> SIP, based on draft-malas-performance-metrics. This memo
    Leslie> would serve as an example of the framework and the PMOL
    Leslie> development process in the IETF.

    Leslie> Discussion of new work proposals is strongly discouraged
    Leslie> under the initial charter of the PMOL WG, except to advise
    Leslie> a protocol development WG when they are evaluating a new
    Leslie> work proposal for related performance metrics.

    Leslie> The PMOL WG will also be guided by a document describing
    Leslie> how memos defining performance metrics are intended to
    Leslie> advance along the IETF Standards track
    Leslie> (draft-bradner-metricstest).

    Leslie> PMOL WG will take advantage of expertise and seek to avoid
    Leslie> overlap with other standards development organizations,
    Leslie> such as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and
    Leslie> others.

    Leslie> Milestones

    Leslie> June 08 SIP Performance Metrics Draft to IESG Review for
    Leslie> consideration as Proposed Standard

    Leslie> Sept 08 PMOL Framework and Guidelines Draft to IESG Review
    Leslie> for consideration as BCP

    Leslie> _______________________________________________
    Leslie> IETF-Announce mailing list [email-masked]
    Leslie> https://www1.ietf.org/mailman/listinfo/ietf-announce

    Leslie> --

    Leslie> -------------------------------------------------------------------
    Leslie> "Reality: Yours to discover."  -- ThinkingCat Leslie
    Leslie> Daigle [email-masked]
    Leslie> -------------------------------------------------------------------

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Date: Mon, 29 Oct 2007 16:04:17 -0400
From: Leslie Daigle <[email-masked]>
To: [email-masked]
Cc: [email-masked], IESG <[email-masked]>
Subject: [PMOL] A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)]

At the Application Performance Metrics BoF in
Chicago, there was a lot of discussion about whether
or how to best capture expertise in the area of performance
metrics for application to IETF protocols.

There seemed to be 2 separate questions on the table:

	1/ how, and
	2/ whether

to generalize a performance metric framework for IETF
protocols.

The proposed PMOL WG addresses the first question -- how
to do it.

On the second point -- the question is really about whether
the IETF community as a whole supports/values performance
metrics and will invest the effort in using any general
framework for existing/new protocols.

I believe the folks that joined the PMOL mailing list after
the APM BoF are assuming that support is there.

I suggest that now is an excellent time for folks _not_
involved in the PMOL effort, but who are working on
protocols that could/should make use of its output, to
voice some opinions on whether or not this approach
is helpful/will be useful to them in future protocols.

Leslie.

-------- Original Message --------
Subject: WG Review: Performance Metrics at Other Layers (pmol)
Date: Mon, 22 Oct 2007 14:15:02 -0400
From: IESG Secretary <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]

A new IETF working group has been proposed in the Operations and
Management Area.  The IESG has not made any determination as yet.
The following draft charter was submitted, and is provided for
informational purposes only.  Please send your comments to the IESG
mailing list ([email-masked]) by October 29.

+++

Performance Metrics at Other Layers (pmol)
==============================================

Current Status: Proposed Working Group

WG Chairs:
TBD

Operations and Management Area:
Dan Romascanu <[email-masked]>
Ronald Bonica <[email-masked]>

Description:

The successful implementation and operation of IP based applications
often depends on some underlying performance measurement infrastructure
that helps service operators or network managers to recognize when
performance is unsatisfactory and identify problems affecting service
quality. Standardized performance metrics add the desirable features of
consistent implementation, interpretation, no comparison.

The IETF has two Working Groups dedicated to the development of
performance metrics however each has strict limitations in their
charters:

- The Benchmarking Methodology WG has addressed a range of networking
technologies and protocols in their long history (such as IEEE 802.3,
ATM, Frame Relay, and Routing Protocols), but the charter strictly
limits their Performance characterizations to the laboratory
environment.

- The IP Performance Metrics WG has the mandate to develop metrics
applicable to the performance of Internet data delivery, but it is
specifically prohibited from developing metrics that characterize
traffic (such as a VoIP stream).

The IETF also has current and completed activities related to the
reporting of application performance metrics (e.g. RAQMON and RTCP XR)
and is also actively involved in the development of reliable transport
protocols which would affect the relationship between IP performance and
application performance.

Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for IP-based protocols and
applications that operate over UDP, TCP, SCTP, DCCP, Forward Error
Correction (FEC) and other robust transport protocols, and that can be
used to characterize traffic on live networks.

The working group will focus on the completion of two RFCs:

1. A PMOL framework and guidelines memo that will describe the necessary
elements of performance metrics of protocols and applications
transported over IETF-specified protocols (such as the formal
definition, purpose, and units of measure) and the various types of
metrics that characterize traffic on live networks (such as metrics
derived from other metrics, possibly on lower layers). The framework
will also address the need to specify the intended audience and the
motivation for the performance metrics. There will also be guidelines
for a performance metric development process that includes entry
criteria for new proposals (how a proposal might be evaluated for
possible endorsement by a protocol development working group), and how
an successful proposal will be developed by PMOL WG in cooperation with
a protocol development WG.

2. A proof-of-concept RFC defining performance metrics for SIP, based on
draft-malas-performance-metrics. This memo would serve as an example of
the framework and the PMOL development process in the IETF.

Discussion of new work proposals is strongly discouraged under the
initial charter of the PMOL WG, except to advise a protocol development
WG when they are evaluating a new work proposal for related performance
metrics.

The PMOL WG will also be guided by a document describing how memos
defining performance metrics are intended to advance along the IETF
Standards track (draft-bradner-metricstest).

PMOL WG will take advantage of expertise and seek to avoid overlap with
other standards development organizations, such as ETSI STQ, ITU-T SG
12, ATIS IIF, ATIS PRQC, and others.

Milestones

June 08 SIP Performance Metrics Draft to IESG Review for consideration
as Proposed Standard

Sept 08 PMOL Framework and Guidelines Draft to IESG Review for
consideration as BCP

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf-announce

-- 

-------------------------------------------------------------------
"Reality:
      Yours to discover."
                                 -- ThinkingCat
Leslie Daigle
[email-masked]
-------------------------------------------------------------------

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Subject: RE: [PMOL] Re: draft-morton-perf-metrics-framework-00
Date: Thu, 25 Oct 2007 12:03:35 -0700
From: "Loki Jorgenson" <[email-masked]>
To: "Brian E Carpenter" <[email-masked]>, "Aamer Akhter (aakhter)" <[email-masked]>
Cc: Nevil Brownlee <[email-masked]>, [email-masked]

Brian/Aamer

"Unreliable" more likely reflects ignorance on our part, more than any
fundamental incoherence between upper and lower layer behaviors.  I
would posit that there exists a coupling between any two layers/metrics
that can be discovered and described.  The work of PMOL could include
the definition of such couplings.

Optimally, there would be an abstracted interface between layers that is
parameterized in terms of existing well-defined quantities - for
example, IPDV or One-way Loss Pattern.  Thus expressions of higher level
metrics (take MOS derived from E-model as an exemplar) can be mapped
explicitly (if not simply) back to underlying packet metrics.

Then the relationship between metrics such as associated variances are
explicitly defined by statistical means and not simply a subject of
off-hand and potentially arbitrary thresholds for "reliability".  Mind,
those coupling description may be quite complex in how they related and
compound underlying metrics.

This might also provide an explicit framework for application or
middleware behaviors to be expressed as translation functions within the
coupling context.  Accounting for adaptive behaviors for "network aware"
applications will otherwise be very challenging.

Of course, I may be just being wishful that such a thing is possible.
It certainly seems non-trivial.

Loki Jorgenson
Apparent Networks
t   604 433 2333 ext 105
m   604 250-4642

-----Original Message-----
From: Brian E Carpenter [mailto:[email-masked]]=20
Sent: Wednesday, October 24, 2007 1:36 PM
To: Aamer Akhter (aakhter)
Cc: Nevil Brownlee; [email-masked]
Subject: Re: [PMOL] Re: draft-morton-perf-metrics-framework-00

On 2007-10-25 01:22, Aamer Akhter (aakhter) wrote:
> Al, et al,
>=20
>>>   When metric Foobar is measured, metrics Barfoo and Foofoo must
>>>   be measured simultaneously between the same endpoints.
>>>   Results for Foobar must be accompanied by the results for Barfoo
>>>   and Foofoo. If the observed variance of Barfoo or Foofoo exceeds
>>>   <level>, the results for Foobar must be labelled as 'unreliable'.
>>>
>=20
> This makes sense.=20
>=20
> I wonder however, about the cases where the application (or L7 entity
etc) that is aware of the underlying disruptive transport and is working
around it.=20
>=20
> Example: the work of digital fountain. I think in these cases, the
metrics for the underlying would be even more important/relevant, but
perhaps the final result shouldn't be labeled unreliable.
>=20
> Thoughts?

My thought is that the variance in the "upper" metric will be very hard
to understand if there is a large variance in the "lower" metrics. There
seem likely to be non-linear effects at work.  If the upper metric is
being used only as an observation (e.g. to check SLA conformance) that
is one thing, but if it's being used to make any kind of prediction or
to analyze the behaviour of the upper layer protocol, I think describing
it as unreliable is justified.

     Brian

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Date: Thu, 25 Oct 2007 09:35:30 +1300
From: Brian E Carpenter <[email-masked]>
Organization: University of Auckland
To: "Aamer Akhter (aakhter)" <[email-masked]>
Subject: Re: [PMOL] Re: draft-morton-perf-metrics-framework-00
Cc: Nevil Brownlee <[email-masked]>, [email-masked]

On 2007-10-25 01:22, Aamer Akhter (aakhter) wrote:
> Al, et al,
> 
>>>   When metric Foobar is measured, metrics Barfoo and Foofoo must
>>>   be measured simultaneously between the same endpoints.
>>>   Results for Foobar must be accompanied by the results for Barfoo
>>>   and Foofoo. If the observed variance of Barfoo or Foofoo exceeds
>>>   <level>, the results for Foobar must be labelled as 'unreliable'.
>>>
> 
> This makes sense. 
> 
> I wonder however, about the cases where the application (or L7 entity etc) that is aware of the underlying disruptive transport and is working around it. 
> 
> Example: the work of digital fountain. I think in these cases, the metrics for the underlying would be even more important/relevant, but perhaps the final result shouldn't be labeled unreliable.
> 
> Thoughts?

My thought is that the variance in the "upper" metric will be very hard to
understand if there is a large variance in the "lower" metrics. There seem
likely to be non-linear effects at work.  If the upper metric is being
used only as an observation (e.g. to check SLA conformance) that is one
thing, but if it's being used to make any kind of prediction or to analyze
the behaviour of the upper layer protocol, I think describing it as
unreliable is justified.

     Brian

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Date: Wed, 24 Oct 2007 17:29:48 -0400
To: "MAK, Leen \(Leen\)" <[email-masked]>
From: Al Morton <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: [PMOL] Re: [OPS-AREA] RE: WG Review: Performance Metrics at Other Layers (pmol)

At 09:50 AM 10/24/2007, MAK, Leen \(Leen\) wrote:
>...The word "also" suggests that this task is understood as a kind of
>secondary task, next to the task to "describe the necessary elements of
>performance metrics of protocols and applications ".
>
>I would like to suggest that the specification of intended audience and
>motivation should be a primary task....

Yes, a clarification for both the charter and the memo itself.

Al
([email-masked] is the best place for further discussion)

At 09:50 AM 10/24/2007, MAK, Leen \(Leen\) wrote:
>All,
>
>I read in the draft charter: "The framework will also address the need
>to specify the intended audience and the motivation for the performance
>metrics."
>
>The word "also" suggests that this task is understood as a kind of
>secondary task, next to the task to "describe the necessary elements of
>performance metrics of protocols and applications ".
>
>I would like to suggest that the specification of intended audience and
>motivation should be a primary task.
>If working bottom-up from the consideration of protocols and their
>properties, it is very easy and very tempting to define all sorts of
>nice and fancy parameters and metrics. However, collecting those
>parameters and storing them and processing them does not come for free.
>
>In order to limit the set of metrics to what is really useful, the
>approach should be the other way. The first question to be answered
>should be: which metrics are needed by the network operators to properly
>maintain the network?
>
>Regards,
>
>Leen Mak.
>

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Subject: RE: [PMOL] Re: draft-morton-perf-metrics-framework-00
Date: Wed, 24 Oct 2007 08:22:10 -0400
From: "Aamer Akhter (aakhter)" <[email-masked]>
To: "Al Morton" <[email-masked]>, "Brian E Carpenter" <[email-masked]>
Cc: Nevil Brownlee <[email-masked]>, [email-masked]

Al, et al,

> >
> >   When metric Foobar is measured, metrics Barfoo and Foofoo must
> >   be measured simultaneously between the same endpoints.
> >   Results for Foobar must be accompanied by the results for Barfoo
> >   and Foofoo. If the observed variance of Barfoo or Foofoo exceeds
> >   <level>, the results for Foobar must be labelled as 'unreliable'.
> >

This makes sense.=20

I wonder however, about the cases where the application (or L7 entity =
etc) that is aware of the underlying disruptive transport and is working =
around it.=20

Example: the work of digital fountain. I think in these cases, the =
metrics for the underlying would be even more important/relevant, but =
perhaps the final result shouldn't be labeled unreliable.

Thoughts?

--=20
Aamer Akhter / [email-masked]
Ent & Commercial Systems, cisco Systems

> -----Original Message-----
> From: Al Morton [mailto:[email-masked]]
> Sent: Tuesday, October 23, 2007 10:28 PM
> To: Brian E Carpenter
> Cc: Nevil Brownlee; [email-masked]
> Subject: [PMOL] Re: draft-morton-perf-metrics-framework-00
>=20
> Hi Brian,
>=20
> The draft is intended for discussion on the <[email-masked]> list.
>=20
> You've raised a worthwhile consideration for the draft, so
> I have cc'd the pmol list for further discussion.
>=20
> regards,
> Al
>=20
> At 06:24 PM 10/23/2007, Brian E Carpenter wrote:
> >Al,
> >
> >I'm not sure where the draft is being discussed, so please feel
> >free to add in any appropriate cc's, including your co-author,
> >whose address is missing in the draft.
> >
> >This comment was triggered by seeing the draft charter for
> >pmol, but I think it really belongs in the document and not in
> >the charter.
> >
> >It's fairly clear for the traditional ippm/bmwg metrics what
> >is meant by "all other things being equal." Once you start
> >discussing metrics for the application level, as in pmol,
> >that's much less clear. I believe that any definition of
> >an upper layer metric should include a discussion of the extent
> >to which lower layer performance must be held stable for the
> >upper layer metric to be of value, and in fact what lower layer
> >metrics MUST be measured simultaneously and what bounds on
> >those metrics must be met. I'd envisage statements of the type
> >
> >   When metric Foobar is measured, metrics Barfoo and Foofoo must
> >   be measured simultaneously between the same endpoints.
> >   Results for Foobar must be accompanied by the results for Barfoo
> >   and Foofoo. If the observed variance of Barfoo or Foofoo exceeds
> >   <level>, the results for Foobar must be labelled as 'unreliable'.
> >
> >As a one word summary, this can be called 'dependencies', but
> >I think the framework needs to be explicit about it.
> >
> >      Brian
> >--
> >Regards
> >    Brian Carpenter
> >    University of Auckland
> >
>=20
>=20
>=20
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pmol

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Date: Tue, 23 Oct 2007 22:28:20 -0400
To: Brian E Carpenter <[email-masked]>
From: Al Morton <[email-masked]>
Cc: Nevil Brownlee <[email-masked]>, [email-masked]
Subject: [PMOL] Re: draft-morton-perf-metrics-framework-00

Hi Brian,

The draft is intended for discussion on the <[email-masked]> list.

You've raised a worthwhile consideration for the draft, so
I have cc'd the pmol list for further discussion.

regards,
Al

At 06:24 PM 10/23/2007, Brian E Carpenter wrote:
>Al,
>
>I'm not sure where the draft is being discussed, so please feel
>free to add in any appropriate cc's, including your co-author,
>whose address is missing in the draft.
>
>This comment was triggered by seeing the draft charter for
>pmol, but I think it really belongs in the document and not in
>the charter.
>
>It's fairly clear for the traditional ippm/bmwg metrics what
>is meant by "all other things being equal." Once you start
>discussing metrics for the application level, as in pmol,
>that's much less clear. I believe that any definition of
>an upper layer metric should include a discussion of the extent
>to which lower layer performance must be held stable for the
>upper layer metric to be of value, and in fact what lower layer
>metrics MUST be measured simultaneously and what bounds on
>those metrics must be met. I'd envisage statements of the type
>
>   When metric Foobar is measured, metrics Barfoo and Foofoo must
>   be measured simultaneously between the same endpoints.
>   Results for Foobar must be accompanied by the results for Barfoo
>   and Foofoo. If the observed variance of Barfoo or Foofoo exceeds
>   <level>, the results for Foobar must be labelled as 'unreliable'.
>
>As a one word summary, this can be called 'dependencies', but
>I think the framework needs to be explicit about it.
>
>      Brian
>--
>Regards
>    Brian Carpenter
>    University of Auckland
>

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Mon, 22 Oct 2007 20:42:07 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>, <[email-masked]>, "IETF IPPM WG" <[email-masked]>, <[email-masked]>
Cc: 
Subject: [PMOL] FW: WG Review: Performance Metrics at Other Layers (pmol) 

=20

=20

-----Original Message-----
From: IESG Secretary [mailto:[email-masked]]=20
Sent: Monday, October 22, 2007 8:15 PM
To: [email-masked]
Subject: WG Review: Performance Metrics at Other Layers (pmol)=20

A new IETF working group has been proposed in the Operations and
Management Area.  The IESG has not made any determination as yet. =20
The following draft charter was submitted, and is provided for
informational purposes only.  Please send your comments to the IESG
mailing list ([email-masked]) by October 29.

+++

Performance Metrics at Other Layers (pmol)
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Current Status: Proposed Working Group

WG Chairs:
TBD

Operations and Management Area:
Dan Romascanu <[email-masked]>
Ronald Bonica <[email-masked]>

Description:

The successful implementation and operation of IP based applications
often depends on some underlying performance measurement infrastructure
that helps service operators or network managers to recognize when
performance is unsatisfactory and identify problems affecting service
quality. Standardized performance metrics add the desirable features of
consistent implementation, interpretation, no comparison.

The IETF has two Working Groups dedicated to the development of
performance metrics however each has strict limitations in their
charters:

- The Benchmarking Methodology WG has addressed a range of networking
technologies and protocols in their long history (such as IEEE 802.3,
ATM, Frame Relay, and Routing Protocols), but the charter strictly
limits their Performance characterizations to the laboratory
environment.

- The IP Performance Metrics WG has the mandate to develop metrics
applicable to the performance of Internet data delivery, but it is
specifically prohibited from developing metrics that characterize
traffic (such as a VoIP stream).

The IETF also has current and completed activities related to the
reporting of application performance metrics (e.g. RAQMON and RTCP XR)
and is also actively involved in the development of reliable transport
protocols which would affect the relationship between IP performance and
application performance.

Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for IP-based protocols and
applications that operate over UDP, TCP, SCTP, DCCP, Forward Error
Correction (FEC) and other robust transport protocols, and that can be
used to characterize traffic on live networks.

The working group will focus on the completion of two RFCs:

1. A PMOL framework and guidelines memo that will describe the necessary
elements of performance metrics of protocols and applications
transported over IETF-specified protocols (such as the formal
definition, purpose, and units of measure) and the various types of
metrics that characterize traffic on live networks (such as metrics
derived from other metrics, possibly on lower layers). The framework
will also address the need to specify the intended audience and the
motivation for the performance metrics. There will also be guidelines
for a performance metric development process that includes entry
criteria for new proposals (how a proposal might be evaluated for
possible endorsement by a protocol development working group), and how
an successful proposal will be developed by PMOL WG in cooperation with
a protocol development WG.

2. A proof-of-concept RFC defining performance metrics for SIP, based on
draft-malas-performance-metrics. This memo would serve as an example of
the framework and the PMOL development process in the IETF.

Discussion of new work proposals is strongly discouraged under the
initial charter of the PMOL WG, except to advise a protocol development
WG when they are evaluating a new work proposal for related performance
metrics.

The PMOL WG will also be guided by a document describing how memos
defining performance metrics are intended to advance along the IETF
Standards track (draft-bradner-metricstest).

PMOL WG will take advantage of expertise and seek to avoid overlap with
other standards development organizations, such as ETSI STQ, ITU-T SG
12, ATIS IIF, ATIS PRQC, and others.

Milestones

June 08 SIP Performance Metrics Draft to IESG Review for consideration
as Proposed Standard

Sept 08 PMOL Framework and Guidelines Draft to IESG Review for
consideration as BCP

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Sun, 21 Oct 2007 16:23:23 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] FW: I-D Action:draft-morton-perf-metrics-framework-00.txt 

This is a multi-part message in MIME format.

------_=_NextPart_001_01C813ED.F150BC75
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

=20
For those of you who may have missed the announcement - the initial pmol
individual submission.=20

Also, you may see in the coming days the IETF Last Call for the WG
charter.=20

Dan

=20

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]=20
Sent: Wednesday, October 17, 2007 2:50 AM
To: [email-masked]
Subject: I-D Action:draft-morton-perf-metrics-framework-00.txt=20

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title           : Framework for Performance Metric Development
	Author(s)       : A. Morton
	Filename        : draft-morton-perf-metrics-framework-00.txt
	Pages           : 8
	Date            : 2007-10-16

This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working group
charters in the IETF.  In this version, the memo refers to a Performance
Metrics Entity, or PM Entity, which may in future be a working group or
directorate or a combination of these two.Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-morton-perf-metrics-framework-
00.txt

To remove yourself from the I-D Announcement list, send a message to
[email-masked] with the word unsubscribe in the body of
the message.
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the
username "anonymous" and a password of your e-mail address. After
logging in, type "cd internet-drafts" and then
	"get draft-morton-perf-metrics-framework-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html or
ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE
/internet-drafts/draft-morton-perf-metrics-framework-00.txt".

NOTE:   The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail
readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

------_=_NextPart_001_01C813ED.F150BC75
Content-Type: application/octet-stream;
	name="ATT1012450.TXT"
Content-Transfer-Encoding: base64
Content-Description: ATT1012450.TXT
Content-Disposition: attachment;
	filename="ATT1012450.TXT"

Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7IGFjY2Vzcy10eXBlPSJtYWlsLXNl
cnZlciI7DQoJc2VydmVyPSJtYWlsc2VydkBpZXRmLm9yZyINCg0KQ29udGVudC1UeXBlOiB0ZXh0
L3BsYWluDQpDb250ZW50LUlEOiA8MjAwNy0xMC0xNjIwNDMxNy5JLURAaWV0Zi5vcmc+DQoNCkVO
Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1tb3J0b24tcGVyZi1tZXRy
aWNzLWZyYW1ld29yay0wMC50eHQNCg==

------_=_NextPart_001_01C813ED.F150BC75
Content-Type: application/octet-stream;
	name="draft-morton-perf-metrics-framework-00.URL"
Content-Transfer-Encoding: base64
Content-Description: draft-morton-perf-metrics-framework-00.URL
Content-Disposition: attachment;
	filename="draft-morton-perf-metrics-framework-00.URL"

W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0
cy9kcmFmdC1tb3J0b24tcGVyZi1tZXRyaWNzLWZyYW1ld29yay0wMC50eHQNCg==

------_=_NextPart_001_01C813ED.F150BC75
Content-Type: text/plain;
	name="ATT1012451.txt"
Content-Transfer-Encoding: base64
Content-Description: ATT1012451.txt
Content-Disposition: attachment;
	filename="ATT1012451.txt"

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCkktRC1Bbm5v
dW5jZSBtYWlsaW5nIGxpc3QNCkktRC1Bbm5vdW5jZUBpZXRmLm9yZw0KaHR0cHM6Ly93d3cxLmll
dGYub3JnL21haWxtYW4vbGlzdGluZm8vaS1kLWFubm91bmNlDQo=

------_=_NextPart_001_01C813ED.F150BC75
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

------_=_NextPart_001_01C813ED.F150BC75--

Date: Tue, 16 Oct 2007 22:41:15 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Fwd: I-D Action:draft-morton-perf-metrics-framework-00.txt 

>A New Internet-Draft is available from the on-line Internet-Drafts 
>directories.
>
>         Title           : Framework for Performance Metric Development
>         Author(s)       : A. Morton, A. Clark
>         Filename        : draft-morton-perf-metrics-framework-00.txt
>         Pages           : 8
>         Date            : 2007-10-16
>
>This memo describes a framework and guidelines for the development of
>performance metrics that are beyond the scope of existing working
>group charters in the IETF.  In this version, the memo refers to a
>Performance Metrics Entity, or PM Entity, which may in future be a
>working group or directorate or a combination of these two.Requirements
>Language
>
>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>document are to be interpreted as described in RFC 2119 [RFC2119].
>
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-morton-perf-metrics-framework-00.txt
>
>To remove yourself from the I-D Announcement list, send a message to
>[email-masked] with the word unsubscribe in the body of
>the message.
>You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
>to change your subscription settings.
>
>Internet-Drafts are also available by anonymous FTP. Login with the
>username "anonymous" and a password of your e-mail address. After
>logging in, type "cd internet-drafts" and then
>         "get draft-morton-perf-metrics-framework-00.txt".
>
>A list of Internet-Drafts directories can be found in
>http://www.ietf.org/shadow.html
>or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>Internet-Drafts can also be obtained by e-mail.
>
>Send a message to:
>         [email-masked].
>In the body type:
>         "FILE /internet-drafts/draft-morton-perf-metrics-framework-00.txt".
>
>NOTE:   The mail server at ietf.org can return the document in
>         MIME-encoded form by using the "mpack" utility.  To use this
>         feature, insert the command "ENCODING mime" before the "FILE"
>         command.  To decode the response(s), you will need "munpack" or
>         a MIME-compliant mail reader.  Different MIME-compliant mail readers
>         exhibit different behavior, especially when dealing with
>         "multipart" MIME messages (i.e. documents which have been split
>         up into multiple messages), so check your local documentation on
>         how to manipulate these messages.
>
>Below is the data which will enable a MIME compliant mail reader
>implementation to automatically retrieve the ASCII version of the
>Internet-Draft.
>
>Content-Type: text/plain
>Content-ID: <[email-masked]>
>
>ENCODING mime
>FILE /internet-drafts/draft-morton-perf-metrics-framework-00.txt
>
>
><ftp://ftp.ietf.org/internet-drafts/draft-morton-perf-metrics-framework-00.txt>
>_______________________________________________
>I-D-Announce mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/i-d-announce

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Date: Tue, 16 Oct 2007 15:01:56 -0400
To: Lars Eggert <[email-masked]>, [email-masked]
From: Al Morton <[email-masked]>
Subject: Re: [PMOL] Suggested updates to proposed charter wording
Cc: 

Hi Lars,

We haven't ignored your comment, but it seems I haven't done
enough to address it.  I'm struggling with how much detail to
add in the charter, and hope some discussion will help refine
what's needed.

When you said, referring to the guidelines and framework deliverable:

>Some of the content of this proposed WG deliverable is what we
>typically want to see already during the BOF proposal phase:
>
>Specifically, the "motivation for work", "how that work fills a
>need and any gap in IETF-chartered work", "motivation for the
>performance metrics" seem pretty relevant already when asking for a
>BOF.

Our intent was to document aspects of the original BOF proposal,
which motivated the work area and identified gaps in IETF coverage,
as an Intro to the guidelines and framework memo. The proposed PMOL
charter just gives a short summary of the BOF proposal text.

Maybe we should add more background in the charter?
from the BoF minutes:
>The BOF introduced the IETF community to the possibility of a 
>generalized activity to define standardized performance metrics. The 
>existence of a growing list of Internet-Drafts on performance metrics
(with community interest in development, but in un-chartered areas)
>supports the need for this activity, and the majority of people 
>present supported the proposition that IETF should be working in 
>these areas. No one objected to any of these proposals. However, 
>many people from beyond the performance community sought 
>clarifications about the scope of the activity. The proposal was 
>deliberately made broad to cover the wide variety of performance 
>work that is currently excluded by the IPPM and BMWG charters.

or maybe we shouldn't mention the Intro material at all?

Last paragraph of your comment:
>Likewise, information on "the various types of metrics that may be
>prepared in this work" seem important to define the scope for the
>planned WG. "Entry criteria for new proposals" is also something
>that should be in the proposed charter already.

The scope of the WG is very limited, and it allows development of only
the two deliverables described: the guidelines and framework memo
(a process for developing performance metrics that don't fall within
other charters and their frameworks) and the SIP metrics memo.
The PMOL WG must develop the process and entry criteria for *additional*
proposals, and it will only work on additional deliverables if
explicitly chartered to do so. A key aspect of the process is that
most proposals will be vetted in a protocol development WG before
reaching PMOL WG.

(maybe that paragraph should be part of the charter)...

We're certainly trying to do something unusual in my short IETF
experience, and appreciate further suggestions or questions to
help clarify what the proposed PMOL can do.

regards,
Al

At 01:51 PM 10/15/2007, Lars Eggert wrote:
>Nothing in the new charter addresses my comment on the previous version:
>
>On 2007-9-26, at 3:09, ext Lars Eggert wrote:
>>On 2007-9-25, at 21:22, ext Alan Clark wrote:
>>>1. A PMOL framework and guidelines memo that outlines the motivation
>>>     for work to define performance metrics for applications
>>>transported
>>>     over IETF-specified protocols, how that work fills a need and
>>>any gap
>>>     in IETF-chartered work. The framework will describe the necessary
>>>     elements of performance metric drafts and the various types of
>>>metrics
>>>     that may be prepared in this work. The framework will also
>>>address the
>>>     need to specify the intended audience and the motivation for the
>>>     performance metrics. There will also be guidelines for a
>>>performance
>>>     metric development process that includes entry criteria for
>>>     new proposals (how a proposal might be evaluated for possible
>>>     endorsement by a protocol development working group), and how a
>>>     successful proposal will be developed by PMOL WG in
>>>cooperation with a
>>>     protocol development WG.
>>
>>Some of the content of this proposed WG deliverable is what we
>>typically want to see already during the BOF proposal phase:
>>
>>Specifically, the "motivation for work", "how that work fills a
>>need and any gap in IETF-chartered work", "motivation for the
>>performance metrics" seem pretty relevant already when asking for a
>>BOF.
>>
>>Likewise, information on "the various types of metrics that may be
>>prepared in this work" seem important to define the scope for the
>>planned WG. "Entry criteria for new proposals" is also something
>>that should be in the proposed charter already.
>>
>>Lars
>
>
>
>_______________________________________________
>PMOL mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/pmol

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

From: Lars Eggert <[email-masked]>
Subject: Re: [PMOL] Suggested updates to proposed charter wording
Date: Mon, 15 Oct 2007 11:51:12 -0600
To: ext Alan Clark <[email-masked]>, [email-masked]
Cc: 

--===============1545476074==
Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-5--584892479;
	protocol="application/pkcs7-signature"

--Apple-Mail-5--584892479
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	delsp=yes;
	format=flowed

Nothing in the new charter addresses my comment on the previous version:

On 2007-9-26, at 3:09, ext Lars Eggert wrote:
> On 2007-9-25, at 21:22, ext Alan Clark wrote:
>> 1. A PMOL framework and guidelines memo that outlines the motivation
>>     for work to define performance metrics for applications  
>> transported
>>     over IETF-specified protocols, how that work fills a need and  
>> any gap
>>     in IETF-chartered work. The framework will describe the necessary
>>     elements of performance metric drafts and the various types of  
>> metrics
>>     that may be prepared in this work. The framework will also  
>> address the
>>     need to specify the intended audience and the motivation for the
>>     performance metrics. There will also be guidelines for a  
>> performance
>>     metric development process that includes entry criteria for
>>     new proposals (how a proposal might be evaluated for possible
>>     endorsement by a protocol development working group), and how a
>>     successful proposal will be developed by PMOL WG in  
>> cooperation with a
>>     protocol development WG.
>
> Some of the content of this proposed WG deliverable is what we  
> typically want to see already during the BOF proposal phase:
>
> Specifically, the "motivation for work", "how that work fills a  
> need and any gap in IETF-chartered work", "motivation for the  
> performance metrics" seem pretty relevant already when asking for a  
> BOF.
>
> Likewise, information on "the various types of metrics that may be  
> prepared in this work" seem important to define the scope for the  
> planned WG. "Entry criteria for new proposals" is also something  
> that should be in the proposed charter already.
>
> Lars

--Apple-Mail-5--584892479
Content-Transfer-Encoding: base64
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Disposition: attachment;
	filename=smime.p7s

--Apple-Mail-5--584892479--

--===============1545476074==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

--===============1545476074==--

Content-class: urn:content-classes:message
Subject: RE: [PMOL] Revised Charter
Date: Mon, 15 Oct 2007 14:42:47 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Cc: 

I would add to the milestones that the framework document will be
considered for BCP and the SIP Performance metrics document as
standards-track.=20

Dan

=20
=20

> -----Original Message-----
> From: Al Morton [mailto:[email-masked]]=20
> Sent: Monday, October 15, 2007 2:32 PM
> To: [email-masked]
> Subject: [PMOL] Revised Charter
>=20
> Folks,
> here's the latest version of the charter text, addressing=20
> most, if not all, comments to date.
> Al
>=20
> Proposed Charter (0.3)
>=20
>=20
> Performance Metrics at Other Layers WG (PMOL)
>=20
> The successful implementation and operation of IP based=20
> applications often depends on some underlying performance=20
> measurement infrastructure that helps service operators or=20
> network managers to recognize when performance is=20
> unsatisfactory and identify problems affecting service=20
> quality.  Standardized performance metrics add the desirable=20
> features of consistent implementation, interpretation, nd comparison.
>=20
> The IETF has two Working Groups dedicated to the development=20
> of performance metrics however each has strict limitations in=20
> their charters:
>=20
>    - The Benchmarking Methodology WG has addressed a range of=20
> networking technologies
>      and protocols in their long history (such as IEEE 802.3,=20
> ATM, Frame Relay, and
>      Routing Protocols), but the charter strictly limits=20
> their performance
>      characterizations to the laboratory environment.
>=20
>    - The IP Performance Metrics WG has the mandate to develop=20
> metrics applicable
>      to the performance of Internet data delivery, but it is=20
> specifically prohibited
>      from developing metrics that characterize traffic (such=20
> as a VoIP stream).
>=20
> The IETF also has current and completed activities related to=20
> the reporting of application performance metrics (e.g. RAQMON=20
> and RTCP XR) and is also actively involved in the development=20
> of reliable transport protocols which would affect the=20
> relationship between IP performance and application performance.
>=20
> Thus there is a gap in the currently chartered coverage of IETF WGs:
> development of performance metrics for IP-based applications=20
> that operate over UDP, TCP, SCTP, DCCP, Forward Error=20
> Correction (FEC) and other robust transport protocols, and=20
> that can be used to characterize traffic on live networks.
>=20
> The working group will focus on the completion of two RFCs:
>=20
> 1. A PMOL framework and guidelines memo that documents the motivation
>      for work to define performance metrics for applications=20
> transported
>      over IETF-specified protocols, and how that work fills a=20
> need and a gap
>      in IETF-chartered work (motivation and gap=20
> identification having been part
>      of the original BOF proposal). The framework will=20
> describe the necessary
>      elements of performance metric drafts (such as the=20
> formal definition,
>      purpose, and units of measure) and the various types of metrics
>      that may be prepared in this work (such as metrics=20
> derived from other
>      metrics, possibly on lower layers). The framework will=20
> also address the
>      need to specify the intended audience and the motivation for the
>      performance metrics. There will also be guidelines for a=20
> performance
>      metric development process that includes entry criteria for
>      new proposals (how a proposal might be evaluated for possible
>      endorsement by a protocol development working group), and how a
>      successful proposal will be developed by PMOL WG in=20
> cooperation with a
>      protocol development WG.
>=20
> 2. A proof-of-concept RFC defining performance metrics for=20
> SIP, based on
>      draft-malas-performance-metrics.  This memo would serve=20
> as an example of
>      the framework and the PMOL development process in the IETF.
>=20
> Discussion of new work proposals is strongly discouraged=20
> under the initial charter of the PMOL WG, except to advise a=20
> protocol development WG when they are evaluating a new work=20
> proposal for related performance metrics.
>=20
> The PMOL WG will also be guided by a document describing how=20
> memos defining performance metrics are intended to advance=20
> along the IETF Standards track (draft-bradner-metricstest).
>=20
> PMOL WG will take advantage of expertise and seek to avoid=20
> overlap with other standards development organizations, such=20
> as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and others.
>=20
> Milestones
> June 08  SIP Performance Metrics Draft to IESG Review Sept 08=20
>  PMOL Framework and Guidelines Draft to IESG Review
>=20
>=20
>=20
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pmol
>=20

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Date: Mon, 15 Oct 2007 08:32:26 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Revised Charter

Folks,
here's the latest version of the charter text,
addressing most, if not all, comments to date.
Al

Proposed Charter (0.3)

Performance Metrics at Other Layers WG (PMOL)

The successful implementation and operation of IP based applications 
often depends
on some underlying performance measurement infrastructure that helps service
operators or network managers to recognize when performance is unsatisfactory
and identify problems affecting service quality.  Standardized performance
metrics add the desirable features of consistent implementation, 
interpretation,
nd comparison.

The IETF has two Working Groups dedicated to the development of 
performance metrics
however each has strict limitations in their charters:

   - The Benchmarking Methodology WG has addressed a range of 
networking technologies
     and protocols in their long history (such as IEEE 802.3, ATM, 
Frame Relay, and
     Routing Protocols), but the charter strictly limits their performance
     characterizations to the laboratory environment.

   - The IP Performance Metrics WG has the mandate to develop metrics 
applicable
     to the performance of Internet data delivery, but it is 
specifically prohibited
     from developing metrics that characterize traffic (such as a VoIP stream).

The IETF also has current and completed activities related to the reporting of
application performance metrics (e.g. RAQMON and RTCP XR) and is also actively
involved in the development of reliable transport protocols which 
would affect the
relationship between IP performance and application performance.

Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for IP-based applications that 
operate over UDP,
TCP, SCTP, DCCP, Forward Error Correction (FEC) and other robust 
transport protocols,
and that can be used to characterize traffic on live networks.

The working group will focus on the completion of two RFCs:

1. A PMOL framework and guidelines memo that documents the motivation
     for work to define performance metrics for applications transported
     over IETF-specified protocols, and how that work fills a need and a gap
     in IETF-chartered work (motivation and gap identification having 
been part
     of the original BOF proposal). The framework will describe the necessary
     elements of performance metric drafts (such as the formal definition,
     purpose, and units of measure) and the various types of metrics
     that may be prepared in this work (such as metrics derived from other
     metrics, possibly on lower layers). The framework will also address the
     need to specify the intended audience and the motivation for the
     performance metrics. There will also be guidelines for a performance
     metric development process that includes entry criteria for
     new proposals (how a proposal might be evaluated for possible
     endorsement by a protocol development working group), and how a
     successful proposal will be developed by PMOL WG in cooperation with a
     protocol development WG.

2. A proof-of-concept RFC defining performance metrics for SIP, based on
     draft-malas-performance-metrics.  This memo would serve as an example of
     the framework and the PMOL development process in the IETF.

Discussion of new work proposals is strongly discouraged under
the initial charter of the PMOL WG, except to advise a protocol development
WG when they are evaluating a new work proposal for related 
performance metrics.

The PMOL WG will also be guided by a document describing how memos
defining performance metrics are intended to advance along the IETF
Standards track (draft-bradner-metricstest).

PMOL WG will take advantage of expertise and seek to avoid overlap with
other standards development organizations, such as ETSI STQ, ITU-T SG 12,
ATIS IIF, ATIS PRQC, and others.

Milestones
June 08  SIP Performance Metrics Draft to IESG Review
Sept 08  PMOL Framework and Guidelines Draft to IESG Review

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Sun, 14 Oct 2007 13:47:07 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>, <[email-masked]>, <[email-masked]>
Cc: 
Subject: [PMOL] FW: REVISED Approval to Post Version -00 Internet-Drafts for the 70th IETF Meeting in Vancouver, BC, Canada 

=20

=20

-----Original Message-----
From: Internet-Drafts Administrator [mailto:[email-masked]]=20
Sent: Friday, October 12, 2007 8:01 PM
To: Working Group Chairs
Cc: [email-masked]
Subject: REVISED Approval to Post Version -00 Internet-Drafts for the
70th IETF Meeting in Vancouver, BC, Canada=20

Dear IETF Working Group Chairs:

In order to expedite the processing of the many version -00 I-Ds that
the Secretariat  receives before an IETF meeting, we ask that you please
notify the Secretariat prior to the initial submission cutoff date of
all version -00 I-Ds that you expect to approve for posting as WG
documents.=20
Please indicate if a new version -00 I-D replaces any existing I-Ds, and
if so, which ones.

Please send the filenames of your approved -00 I-Ds to
[email-masked], or use the "Initial Version WG Draft Approval
Tool" https://datatracker.ietf.org/cgi-bin/wg/wg_init_rev_approval.cgi
to pre-approve drafts.  We would appreciate receiving your list of
approved drafts five (5) business days prior to the cutoff date for -00
submissions, or by Monday, November 5th at 9:00 AM ET (14:00 UTC/GMT)
for the 70th IETF Meeting.  If you send your list by e-mail, then please
include the word "Approved I-Ds" in the "Subject" field.  This procedure
will help to ensure that version -00 I-Ds are posted in a timely manner,
allowing more time for review by the public.

Thank you for your cooperation in this matter.

The Internet-Drafts Administrator

FYI: The Internet-Draft cutoff dates as well as other significant dates
for the 70th IETF Meeting can be found at
http://www3.ietf.org/meetings/70-cutoff_dates.html.

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol