Subject: Re: [PMOL] Fwd: RE: WGLC on pmol-metrics-framework-01
From: Benoit Claise <[email-masked]>
To: Al Morton <[email-masked]>, Loki Jorgenson <[email-masked]>
Cc: [email-masked]
Date: Mon, 26 Oct 2009 15:17:34 +0100
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

Loki,

Thanks for your feedback.

I have added the statement that this framework is valid for both active 
and passive measurements.

All the editorial comments have been taken into account.

Regards, Benoit.
> Forwarded at Loki's request...
>
>> X-VirusChecked: Checked
>> X-Env-Sender: [email-masked]
>> X-Msg-Ref: server-13.tower-121.messagelabs.com!1228755277!24935304!1
>> X-StarScan-Version: 5.5.12.14.2; banners=-,-,-
>> X-Originating-IP: [209.139.228.52]
>> X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG
>> X-AuditID: ac108108-00000e4c00000774-d5-493d514d1d00
>> Subject: RE: [PMOL] WGLC on pmol-metrics-framework-01
>> Date: Mon, 8 Dec 2008 08:54:35 -0800
>> X-MS-Has-Attach:
>> X-MS-TNEF-Correlator:
>> thread-topic: [PMOL] WGLC on pmol-metrics-framework-01
>> thread-index: AclUuJeWbGY6AyloRw+sWjN4K8ltvABneGJgAL/HSyA=
>> From: "Loki Jorgenson" <[email-masked]>
>> To: "Al Morton" <[email-masked]>
>> X-Brightmail-Tracker: AAAAAA==
>>
>> Al - any reason why my post to PMOL did not make it onto the list?
>>
>> Loki Jorgenson
>> Apparent Networks
>> t   604 433 2333 ext 105
>> m   604 250-4642
>>
>> -----Original Message-----
>> From: Loki Jorgenson
>> Sent: Thursday, December 04, 2008 16:49
>> To: [email-masked]
>> Cc: Alan Clark
>> Subject: RE: [PMOL] WGLC on pmol-metrics-framework-01
>>
>> Pardon my late comments as I have been unavailable for PMOL
>> participation as of late.
>>
>> One request for clarification as well as some tidying of prose.
>>
>> 1) Is there any value in defining the relationship of this framework to
>> active or passive measurement methodologies?
>>
>> Originally I had thought not.  It appeared that this document was
>> intended to be high-level and agnostic to active or passive
>> methodologies, and therefore said nothing differentiate between the two.
>>
>>
>> However, there is a reference in Section 6 (Security Considerations)
>> that states that "security considerations that apply to any active
>> measurements of live networks are relevant here" with a pointer to
>> IPPM-developed RFC4656, OWAMP.  IPPM thus far limits itself to active
>> methodologies.  This very mildly suggests that this framework is
>> specific to active techniques.
>>
>> In Purpose and Scope, the reference is "for... metrics... that can be
>> used to characterize traffic on live networks and services" - which
>> sounds distinctly passive, insofar as "traffic" suggests applications
>> and not active probing.
>>
>> And, various cross-references to other IPPM drafts and RFCs have been
>> directly incorporated (e.g. compagg in at least Section 3.3).  However,
>> examples to the Telchemy burst-loss algorithm and RFC 4546 may be
>> applied to either passive or active.
>>
>> Further, if passive is included, then security considerations specific
>> to passive should apply in Section 6 as well (e.g. anonymization).  I
>> don't have a handy sample reference for that.
>>
>> Not to make more of it than necessary, but maybe it is worth addressing
>> explicitly.  Perhaps a sentence or two within the Purpose and Scope
>> section declaring its applicability with regard to common measurement
>> methodologies.
>>
>> 2) Awkward sentence:
>>
>> Section 3.4.2 (v)
>>
>> "Short intervals or frequent sampling provides a richer source of
>> information that can be helpful in assessing application performance
>> however can lead to excessive measurement data."
>>
>> It is comprehensible but somewhat awkward.  Suggest a couple of changes:
>>
>> "A SHORT SAMPLING INTERVAL or frequent sampling provides a richer source
>> of information that MAY be helpful in assessing application performance
>> BUT MAY ALSO lead to excessive measurement data."
>>
>> (Capitals indicate changes, not use of capitals in the document - e.g.
>> MAY here is not the usual IETF directive convention)
>>
>> 3) Ambiguous sentence:
>>
>> Section 3.4.2 (v)
>>
>> "Long measurement or sampling intervals reduce that amount of reported
>> and collected data however may be insufficient to truly understand
>> application performance or service quality if this varies with time."
>>
>> Semantically challenged although more or less readable.  Suggest:
>>
>> "Long measurement or sampling intervals reduce THE amount of reported
>> and collected data SUCH THAT IT may be insufficient to ___ understand
>> application performance or service quality INSOFAR AS THE MEASURED
>> QUANTITY MAY vary SIGNIFICANTLY with time."
>>
>> 4) Awkward sentence:
>>
>> Section 3.4.3 (iv)
>>
>> "For example, if the metric is a short term running average packet delay
>> variation (e.g.  PPDV as defined in RFC3550) however this value is
>> reported at intervals of 6-10 seconds this results in a sampling model
>> which may have limited accuracy if packet delay variation is
>> non-stationary."
>>
>> A very long sentence with missing or awkward conjunction.  Suggest:
>>
>> "For example, if the metric is a short term running average packet delay
>> variation (e.g.  PPDV as defined in RFC3550) AND (YET) this value is
>> reported at intervals of 6-10 seconds, THE RESULTING MEASUREMENT may
>> have limited accuracy WHEN packet delay variation is non-stationary."
>>
>> 5) Typographic error:
>>
>> Section 3.4.5 (Measurement Timing:)
>>
>> "... due to time-of-day load network load changes."
>>
>> Suggest:
>>
>> "... due to time-of-day changes in network load."
>>
>>
>> 6) Inconsistent notation:
>>
>> Section 3.5.1
>>
>> "... a 10 percent variation ..." (two instances)
>>
>> Could using number word and "percent" for all instances or numeric value
>> and '%'.  Suggest:
>>
>> "... a 10% variation ..." (two instances)
>>
>>
>> End of Comments
>>
>> ------------------------------------------------------------------------
>> -------------
>>
>> Message: 1
>> Date: Tue, 02 Dec 2008 10:11:00 -0500
>> From: Al Morton <[email-masked]>
>> Subject: [PMOL] WGLC on pmol-metrics-framework-01
>> To: [email-masked]
>> Cc: [email-masked], [email-masked]
>>
>> PMOL WG,
>> cc IPPM and BMWG WGs,
>>
>> This message begins a WG Last Call on the following draft:
>>
>>         Title           : Framework for Performance Metric Development
>>         Author(s)       : A. Clark
>>         Filename        : draft-ietf-pmol-metrics-framework-01.txt
>>         Pages           : 14
>>         Date            : 2008-11-02
>>
>>
>
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol

Date: Mon, 26 Oct 2009 15:15:10 +0100
From: Benoit Claise <[email-masked]>
To: "Christou, Christos [USA]" <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] application performance metrics

This is a multi-part message in MIME format.
--------------000503030304070300050503
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Chris,

In the new framework draft version I just posted, I inserted definitions 
for QoE, QoS, and I have a place holder for application performance 
metric, for which I'm after a generic definition.
See if this works for you.

Regards, Benoit.
> Hello,
>  
> Having reviewed the work of this working group and listening to the 
> presentation at the Ops Area open meeting in Minneapolis, I would like 
> to understand from WG participants whether there is interest in 
> expanding the work of this WG to define performance metrics for 
> different application types? 
>  
> In discussing this with Al and others, I wanted to see if participants 
> think this is needed, if it is too difficult, out of scope for the 
> IETF, if the appropriate metrics have already been defined, etc.  For 
> example, ITU-T SG 12 has definied G.1080 and G.1081 to define 
> different "QoE" metrics--is this sufficient? 
>  
> It seems like there has been much work over the years on how to 
> measure MOS.  However, what is the parameter that is used for other 
> applications?  If this has previously been defined, please do include 
> a pointer...
>  
> Thanks--just wanted to gauge the list's thoughts on this,
>  
> Chris
> ------------------------------------------------------------------------
>
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol
>   

--------------000503030304070300050503
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Chris,<br>
<br>
In the new framework draft version I just posted, I inserted
definitions for QoE, QoS, and I have a place holder for application
performance metric, for which I'm after a generic definition.<br>
See if this works for you.<br>
<br>
Regards, Benoit.<br>
<blockquote
 cite="mid:[email-masked]"
 type="cite">
  <meta http-equiv="Content-Type" content="text/html; ">
  <meta content="MSHTML 6.00.2900.3429" name="GENERATOR">
  <div>
  <div><span class="730173919-19112008"><font face="Arial" size="2">Hello,</font></span></div>
  <div><span class="730173919-19112008"></span>&nbsp;</div>
  <div><span class="730173919-19112008"><font face="Arial" size="2">Having
reviewed the work of this working group and listening to the
presentation at the Ops Area open meeting&nbsp;<span
 class="918425415-01122008">in Minneapolis</span>, I would like to
understand from WG participants whether there is interest in expanding
the work of this WG to define performance metrics for different
application types?&nbsp; </font></span></div>
  <div><span class="730173919-19112008"></span>&nbsp;</div>
  <div><span class="730173919-19112008"><font face="Arial" size="2">In
discussing this with Al and others, I wanted to see if participants
think this is needed, if it is too difficult, out of scope for the
IETF, if the appropriate metrics have already been defined, etc.&nbsp; For
example, ITU-T SG 12 has definied G.1080 and G.1081 to define different
"QoE" metrics--is this sufficient?&nbsp; </font></span></div>
  <div><span class="730173919-19112008"></span>&nbsp;</div>
  <div><span class="730173919-19112008"><font face="Arial" size="2">It&nbsp;seems
like there has been much work over the years on how to measure MOS.&nbsp;
However, what is the parameter that is used for other applications?&nbsp; If
this has previously been defined, please do include a pointer...</font></span></div>
  <div><span class="730173919-19112008"></span>&nbsp;</div>
  <div><span class="730173919-19112008"><font face="Arial" size="2">Thanks--just
wanted to gauge the list<span class="918425415-01122008">'</span>s
thoughts on this,</font></span></div>
  <div><span class="730173919-19112008"></span>&nbsp;</div>
  <div><span class="730173919-19112008"><font face="Arial" size="2">Chris</font></span></div>
  </div>
  <pre wrap="">
<hr size="4" width="90%">
_______________________________________________
PMOL mailing list
<a class="moz-txt-link-abbreviated" href="mailto:[email-masked]">[email-masked]</a>
<a class="moz-txt-link-freetext" href="https://www.ietf.org/mailman/listinfo/pmol">https://www.ietf.org/mailman/listinfo/pmol</a>
  </pre>
</blockquote>
<br>
</body>
</html>

--------------000503030304070300050503--

Date: Mon, 26 Oct 2009 15:20:44 +0100
From: Benoit Claise <[email-masked]>
To: "Fardid, Reza" <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] [ippm] WGLC on pmol-metrics-framework-01

Hi Fardid,

Thanks for your feedback.
> Hi,
>
> I have a number of comments on the PMOL framework draft:
>
> 1.1 Since RAQMON is mentioned, adding an Informative reference to its
> framework is appropriate.
>   
Done.
> 3.3.2 Since Index is a derived metric, it is best to specify how its
> components contribute to its measurement beyond a function definition.
> An index composed of N orthogonal metrics all of which are measurable
> has more "usefulness" than an index whose measurement is based on a
> small subset of these N metrics.
>   
Could you please provide some text for this.
> 3.4.2 (vi) Does measurement point define the measurement domain? If not,
> then measurement domain (intra-, inter-) is another factor to consider.
> Security restrictions may prevent inter-domain measurement of certain
> metrics.
>   
Done.
> 3.4.3 Are both active and passive measurement types within scope? If so,
> then measurement type is another factor to consider.
>   
I realize I forgot this one is the version I just posted.
I'll write this down in my personal to do list.
> 3.5 Some metrics and their measurements may be affected by middleboxes.
> I can think of the following:
>
> 3.5.3 Middlebox presence
>
> Presence of a middlebox, e.g., proxy, NAT, redirect server, session
> border controller (SBC), and application layer gateway (ALG) may add
> variability to or restrict the scope of measurements of a metric.  For
> example, an SBC that does not process RTP loopback packets may block or
> locally terminate this traffic rather then pass it through to its
> target.
>   
Inserted.

Regards, Benoit.
> Thanks,
> Reza Fardid
>
>
>
>
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Al Morton
> Sent: Tuesday, December 02, 2008 7:11 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: [ippm] WGLC on pmol-metrics-framework-01
>
> PMOL WG,
> cc IPPM and BMWG WGs,
>
> This message begins a WG Last Call on the following draft:
>
>         Title           : Framework for Performance Metric Development
> 	Author(s)       : A. Clark
> 	Filename        : draft-ietf-pmol-metrics-framework-01.txt
> 	Pages           : 14
> 	Date            : 2008-11-02
>
> 	
> This memo describes a framework and guidelines for the development of
> performance metrics that are beyond the scope of existing working
> group charters in the IETF.  In this version, the memo refers to a
> Performance Metrics Entity, or PM Entity, which may in future be a
> working group or directorate or a combination of these two.
>
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-01
> .txt
>
> There has been discussion of this draft for over a year,
> mostly on the PMOL list and at meetings.  The Author believes he
> has addressed all comments to date.
>
> Please weigh-in on whether or not this Draft should be forwarded
> to the Area Directors for publication as a Standards Track RFC.
> Send your comments to the PMOL list or the co-chairs.
>
> The Last Call will be open till January 2, 2009.
>
> thanks for your review and comments,
> Al Morton
> co-chair, PMOL WG
>
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol
>   

Date: Mon, 26 Oct 2009 15:11:03 +0100
From: Benoit Claise <[email-masked]>
To: Al Morton <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] Suggestions for the framework draft

Al,

Thanks for your suggestion.
All inserted.

Regards, Benoit.
> PMOL,
>
> While reviewing the SIP metrics draft, I found a few places
> where the framework draft needed further development.
>
> I suggest some text in several sections below. I'm not
> exactly sure where most of them fit in the outline, but
> section 4.2 is strawman text for an empty section.
>
> Al
> (as a participant)
>
> 3.x.  Organization of Results
>
>     The IPPM Framework [RFC2330] organizes the results of metrics into
>     three related notions:
>
>     1.  singleton, an elementary instance, or "atomic" value
>
>     2.  sample, a set of singletons with some common properties and some
>         varying properties.
>
>     3.  statistic, a value derived from a sample through deterministic
>         calculation, such as the mean.
>
>     Many metrics can use this organization for the results, with or
>     without the term names used by IPPM working group.  Section 11 of
>     [RFC2330] should consulted for further details.
>
> 3.y.  Parameters, the variables of a metric
>
>     Metrics are completely defined when all options and input variables
>     have been identified and considered.  These variables are sometimes
>     left unspecified in a metric definition, and their general name
>     indicates that the user must set them and report them with the
>     results.  Such variables are called "parameters" in the IPPM metric
>     template.  The scope of the metric, the time at which it was
>     conducted, the settings for timers and the thresholds for counters
>     are all examples of parameters.
>
>     All memos defining performance metrics SHOULD identify ALL key
>     parameters for each metric.
>
> 3.z.  Packet Measurement Details
>
>     Another key aspect of the IPPM Framework [RFC2330] is the discussion
>     of "wire time".  Internet hosts require physical resources to observe
>     packet traffic, and they can contribute to the performance observed.
>     Further, the observations SHOULD be related to some instant of time
>     (such as a specific bit of a packet), because a packet contains many
>     bits and exists on physical media long enough to introduce some
>     ambiguity.  Section 10.2 of [RFC2330] should be consulted.
>
> 3.a.  Identifying and Categorizing the Audience
>
>     Many of the aspects of metric definition and reporting, even the
>     selection or determination of the essential metrics, depend on who
>     will use the results, and for what purpose.  The question, "How will
>     the results be used?" usually yields important factors to consider
>     when developing performance metrics.
>
>     All memos defining performance metrics SHOULD identify the primary
>     audience and its associated requirements.  The audience can influence
>     both the definition of metrics and the methods of measurement.
>
>     The key areas of variation between different metric users include:
>
>     o  Suitability of passive measurements of live traffic, or active
>        measurements using dedicated traffic
>
>     o  Measurement in laboratory environment, or on a network of deployed
>        devices
>
>     o  Accuracy of the results
>
>     o  Access to measurement points and configuration information
>
>     o  Measurement topology (point-to-point, point-to-multipoint)
>
>     o  Scale of the measurement system
>
>     o  Measurements conducted on-demand, or continuously
>
>     o  Required reporting formats
>
>
>
>
> 4.2.  Proposal Approval
>
>     The process depends on the form that the PM Entity ultimately takes,
>     Directorate or working group.
>
>     In all cases, the proposal will need to achieve consensus, in the
>     corresponding protocol development working group (or alternatively,
>     an "Area" working group with broad charter), that there is interest
>     and a need for the work.
>
>     IF the PM Entity is a Directorate,
>
>     THEN Approval SHOULD include the following steps
>
>     o  consultation with the PM Directorate, using this framework memo
>
>     o  consultation with Area Director(s)
>
>     o  and possibly IESG approval of a new or revised charter for the
>        working group
>
>     IF the PM Entity is a Working Group, and the protocol development
>     working group decides to take up the work under its charter,
>
>     THEN the approval is the same as the PM Directorate steps above, with
>     the possible additional assignment of a PM Advisor for the work item.
>
>     IF the PM Entity is a Working Group, and the protocol development
>     working group decides it does not have sufficient expertise to
>     take-up the work, or the proposal falls outside the current charter,
>
>     THEN
>
>     Approval SHOULD include the following steps
>
>     o  identification of protocol expertise to support metric development
>
>     o  consensus in the PM working group that there is interest and a
>        need for the work, and that a memo conforming to this framework
>        can be successfully developed
>
>     o  consultation with Area Director(s)
>
>     o  IESG approval of a revised charter for the PM working group
>
>
> _______________________________________________
> PMOL mailing list
> [email-masked]
> http://www.ietf.org/mailman/listinfo/pmol
>   

From: [email-masked]
To: [email-masked]
Date: Mon, 26 Oct 2009 07:15:02 -0700 (PDT)
Cc: [email-masked]
Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-03.txt

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF.

	Title           : Framework for Performance Metric Development
	Author(s)       : A. Clark, B. Claise
	Filename        : draft-ietf-pmol-metrics-framework-03.txt
	Pages           : 22
	Date            : 2009-10-26

This document describes a framework and a process for developing
performance metrics for IP-based applications that operate over
reliable or datagram transport protocols, and that can be used to
characterize traffic on live networks and services.  The framework
refers to a Performance Metrics Entity, or PM Entity, which may in
future be a working group or directorate or a combination of these
two.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-03.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Message/External-body;
	name="draft-ietf-pmol-metrics-framework-03.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID: <[email-masked]>

--NextPart--

Date: Mon, 26 Oct 2009 15:10:31 +0100
From: Benoit Claise <[email-masked]>
To: [email-masked]
Subject: [PMOL] [Fwd: New Version Notification for draft-ietf-pmol-metrics-framework-03]

This is a multi-part message in MIME format.
--------------040505020504070300000207
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Dear all,

As agreed with Al Morton, I took the editor role in order to finish this 
document.
I have integrated many suggested improvements compared to the previous 
version (which expired).
As you can see there is a TO DO list, and some EDITOR'S NOTES throughout 
the document.
All comments, feedbacks, improvements, text are welcome.

Regards, Benoit.

-------- Original Message --------
Subject: 	New Version Notification for draft-ietf-pmol-metrics-framework-03
Date: 	Mon, 26 Oct 2009 07:05:41 -0700 (PDT)
From: 	IETF I-D Submission Tool <[email-masked]>
To: 	[email-masked]
CC: 	[email-masked]

A new version of I-D, draft-ietf-pmol-metrics-framework-03.txt has been successfuly submitted by Benoit Claise and posted to the IETF repository.

Filename:	 draft-ietf-pmol-metrics-framework
Revision:	 03
Title:		 Framework for Performance Metric Development
Creation_date:	 2009-10-26
WG ID:		 pmol
Number_of_pages: 22

Abstract:
This document describes a framework and a process for developing
performance metrics for IP-based applications that operate over
reliable or datagram transport protocols, and that can be used to
characterize traffic on live networks and services.  The framework
refers to a Performance Metrics Entity, or PM Entity, which may in
future be a working group or directorate or a combination of these
two.

The IETF Secretariat.

--------------040505020504070300000207
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
</head>
<body bgcolor="#ffffff" text="#000000">
Dear all,<br>
<br>
As agreed with Al Morton, I took the editor role in order to finish
this document.<br>
I have integrated many suggested improvements compared to the previous
version (which expired).<br>
As you can see there is a TO DO list, and some EDITOR'S NOTES
throughout the document.<br>
All comments, feedbacks, improvements, text are welcome.<br>
<br>
Regards, Benoit.<br>
<br>
-------- Original Message --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
 cellspacing="0">
  <tbody>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">Subject: </th>
      <td>New Version Notification for
draft-ietf-pmol-metrics-framework-03</td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">Date: </th>
      <td>Mon, 26 Oct 2009 07:05:41 -0700 (PDT)</td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">From: </th>
      <td>IETF I-D Submission Tool <a class="moz-txt-link-rfc2396E" href="mailto:[email-masked]">&lt;[email-masked]&gt;</a></td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">To: </th>
      <td><a class="moz-txt-link-abbreviated" href="mailto:[email-masked]">[email-masked]</a></td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">CC: </th>
      <td><a class="moz-txt-link-abbreviated" href="mailto:[email-masked]">[email-masked]</a></td>
    </tr>
  </tbody>
</table>
<br>
<br>
<pre>A new version of I-D, draft-ietf-pmol-metrics-framework-03.txt has been successfuly submitted by Benoit Claise and posted to the IETF repository.

Filename:	 draft-ietf-pmol-metrics-framework
Revision:	 03
Title:		 Framework for Performance Metric Development
Creation_date:	 2009-10-26
WG ID:		 pmol
Number_of_pages: 22

Abstract:
This document describes a framework and a process for developing
performance metrics for IP-based applications that operate over
reliable or datagram transport protocols, and that can be used to
characterize traffic on live networks and services.  The framework
refers to a Performance Metrics Entity, or PM Entity, which may in
future be a working group or directorate or a combination of these
two.

The IETF Secretariat.

</pre>
</body>
</html>

--------------040505020504070300000207--

From: Robert Sparks <[email-masked]>
To: Al Morton <[email-masked]>
Date: Tue, 6 Oct 2009 10:38:50 -0500
Cc: [email-masked]
Subject: Re: [PMOL] Several questions/suggestions from my review of draft-ietf-pmol-sip-perf-metrics-04

Thanks Al -

I'm going to enter a discuss pointing to my original message in the  
PMOL archives. The tracker will send email to you and the authors.
I wanted to let you know I had seen this and that the message the  
tracker sends you is not a prod.

RjS

On Sep 30, 2009, at 6:49 AM, Al Morton wrote:

> Robert,
>
> thanks for your questions & suggestions. Daryl and I will get
> together to address them, and will be in touch soon.
>
> Al (as co-author)
>
> +++++++++++++++++
>
> PMOL WG,
> If you have strong opinions about any of the areas
> that Robert identified in his message, feel free to
> post to the list on the topic(s)
> (using a new and descriptive subject line, please).
>
> thanks,
> Al (as pmol co-chair)
>
> At 02:31 PM 9/29/2009, Robert Sparks wrote:
>> Hi All -
>>
>> I have several concerns about draft-ietf-pmol-sip-perf-metrics that I
>> would like to discuss.
>> I've asked for a dedicated RAI-review of this document, so there may
>> be additional comments
>> later, but I wanted to get these to you now so we can start working
>> through them.
>>
>> These comments are more-or-less in document order, with a couple of
>> nits moved to the end.
>> I've numbered them to help split responses into threads later. When
>> replying to just one of the
>> items below, please change the subject line to indicate what you're
>> replying to.
>>
>> Thanks,
>>
>> RjS
>

From: "Dale Worley" <[email-masked]>
To: [email-masked]
Organization: Nortel Networks
Date: Fri, 02 Oct 2009 22:33:20 -0400
Cc: [email-masked], [email-masked], Mary Barnes <[email-masked]>, [email-masked]
Subject: [PMOL] RAI-ART review of draft-ietf-pmol-sip-perf-metrics-04 by Dale R.	Worley

RAI-ART review of draft-ietf-pmol-sip-perf-metrics-04
Dale R. Worley

I am the assigned RAI-ART reviewer for draft-ietf-pmol-sip-perf-metrics-04.txt.

For background on RAI-ART, please see the FAQ at
<http://www.softarmor.com/rai/art/FAQ.html>.

Please resolve these comments along with any other Last Call comments
you may receive.

This draft is on the right track but has open issues, described in the review.

My comments are divided into groups of varying significance:

- Situations where the draft appears to have omitted measurements that
  would be of value, and which appear to be of the same category as
  measurements that are included in the draft.  However, this omission
  may have been deliberate due to the intended scope of the draft, in
  which case the intended limitation should be described (and perhaps
  justified) in the "Scope" section.

- Deficiencies in the exposition of how various timing measurements
  are to be taken.  These may also mask technical issues about how the
  measurements are intended to be taken.

- Editorial issues, mostly involving aligning terminology with the
  common usage in SIP discussions.

The items in these groups are numbered with discontinuous sets of
numbers.

Situations where the draft appears to have omitted measurements that
would be of value, and which appear to be of the same category as
measurements that are included in the draft.  However, this omission
may have been deliberate due to the intended scope of the draft, in
which case the intended limitation should be described (and perhaps
justified) in the "Scope" section.

100. Is the intention to restrict attention to signaling (SIP) alone?
In our experience, performance problems first come to users' attention
in media (RTP), and any environment with tolerable media performance
has more than adequate signaling performance.

101. The measurements appear to be designed to closely parallel
performance metrics of TDM telephone systems.  This may be
intentional, but this draft omits a number of measurements that do not
closely parallel TDM performance metrics, but are nonetheless
important for the performance of SIP systems, even when limiting
attention to "traditional telephone" usage.  (See later items for
specifics.)

102. Many of the metrics appear to form natural triples:

- Average delay when the operation (of a particular class) was successful.
- Average delay when the operation was unsuccessful.
- Percentage of operations of the class which were successful.

The metrics scheme would be clarified and made more systematic if this
grouping was defined as an overall property of the metric scheme and
then applied as such to the various classes of operations:
registration request, session request, session establishment, session
disconnect, etc.

E.g., although "Failed session completion delay" (average
BYE-with-failure response delay) is defined, the *percentage* of
failed session completions is not defined, despite that that metric
should be very low in any well-functioning network, and thus can
provide a valuable performance indicator.

103. Some metrics distinguish between "failure in the network" and
"failure at the destination user agent"; e.g., "Session Defects
Ratio".  Beware that distinguishing between these two cases is very
difficult and cannot easily be done by listing response codes for the
two cases.  (I've been developing an I-D for a couple of years on how
to deal with this problem -- I have yet to devise a good solution.)

It may also be desirable to apply this distinction to other operations
other than initial INVITEs -- this would effectively add further
metrics to the "natural triple" described in item 102, as it divides
the class of failures into two sub-classes.

104. In regard to "telephone calls" (INVITE-initiated sessions), there
are three metrics described (each of which gives rise to a triple of
metrics):

- Session request (INVITE to 180)
- Session setup (INVITE to final response)
- Session disconnect (BYE to final response)

In addition, there is a metric for REGISTER requests.

But there are a number of additional operations whose performance is
important to the performance of SIP signaling, and which may differ
greatly from the performance of the above operations (and for which
the users' expectations may be very different from the above
operations):

- re-INVITEs
  (These are particularly important as re-INVITE is used to implement
  on-hold and off-hold operations, and users expect those operations
  to complete much quicker than initial INVITEs.)
- UPDATE (if it is used in practice)
- initial SUBSCRIBE (for dialog events)
- re-SUBSCRIBE
- NOTIFY

Several metrics might be defined regarding REFER requests, which are
used to implement blind and consultative transfers.

In addition, MESSAGE requests can be sent either out-of-dialog or
within-dialog, and one expects the performance of the two cases to be
quite different, so there should be separate metrics for the two
cases.

105. Similar to the problem with detecting "failure in the network",
determining when session request is finished (that is, ringing starts)
is difficult to specify.  A 180 response is clear indication that a
user has been notified, but other 18x responses may be considered as
successful setup in some situations.  E.g., a 182 Queued message may
be considered "end of session request" if one is concerned about the
performance of the network, although it does not indicate the start of
useful communication from the user's point of view.

106. In regard to issues 105 and 103, it may be necessary to allow
that metrics may be "parametrized" by attaching a specification of
which response codes are treated as having which meaning.  In any
case, allowance needs to be made that experience with the metrics may
show that the various specified sets of response codes need to be
modified to maximize the usefulness of the metrics.

Deficiencies in the exposition of how various timing measurements are
to be taken.  However, these may mask technical issues about how the
measurements are intended to be taken.

200. Section 3 of the draft gives a nice standard for how the time
from sending a request to receiving a response is to be measured ("T1
to T4"), including within it a standard for how to measure the time of
sending and receiving ("first bit sent" vs. "last bit received").
However, in many places (e.g., section 4.1) these standards are not
referenced, but rather they are restated.  This leads the reader to
have to check each metric definition to see if it is defined in the
same way as described in section 3, and to wonder if all of the
metrics are defined in the same way.  Better would be to have section
3 note explicitly that all metrics use this definition of time delay.

201. Metrics involving INVITE should explicitly note that ACK is not
considered as part of the time delay; all time delays are measured
from sending the INVITE to receiving its response.  Or rather, there
should be an overall declaration of this standard in section 3.

202. The discussion of "Successful session duration SDT" shows some
examples but does not clearly indicate how all four cases are to be
handled.  (The cases are the combinations of:  measurement at the
originating end vs. measurement at the terminating end, sending BYE
vs. receiving BYE.)  There is also some confusion regarding T1 and T4,
which are specified in multiple different ways in this section.

203. Consideration should be made in the various definitions of calls
that are timed-out by the session-timer mechanism or other
session-keepalive mechanisms.  (Unless it is known that the intended
networks will not use session-keepalives, or that session-keepalive
failures will not cause UAs to see differing signaling flows.)

204. As Robert notes, the "Hops per Request" measurement is unlikely
to be useful for out-of-dialog requests, because it does not capture
redirections handled by proxies and failed forks, and so is poorly
correlated with the amount of work needed to deliver a request to its
destination.  However, this metric may be useful for in-dialog
requests, as there is usually no redirection and (AFAIK) B2BUAs
maintain the Max-Forwards value.

Editorial issues, mostly involving aligning terminology with the
common usage in SIP discussions.

300. Many of the metrics appear to be intended to report the average
of a quantity that is measured for many executions of an operation.
But most of these metrics do not state that averaging is to be done,
they describe in detail how a delay quantity is to be measured, but do
not mention that averaging is to be done.  (E.g., the word "average"
appears only twice in the draft.)

301. The term "session" seems to always be used to describe what is
called a "dialog" in the SIP world, seeing has how the media (RTP)
session is never discussed.  Is this a telecom usage, or should it be
replaced with "dialog"?

302. As Robert notes, T1 and T4 are used in ways that conflict with
the use of those symbols in RFC 3261.  They also lead one to wonder
what happened to T2 and T3.  But perhaps these symbols are a reference
to a terminology used in TDM performance measurement and should be
retained due to that standardization.

303. In measurements regarding a single request and its response(s),
the roles UAC and UAS are well-defined.  But in session-duration
measurements, the roles of UAC and UAS are defined only within
individual request-response transactions.  In that case, a different
terminology should be used.  (E.g., "originator" vs. "terminator"? --
Surely the telecom world has words for this.)