Subject: RE: [midcom] Comments on protocol evaluation draft

From: "Mary Barnes" <[email-masked]>
Date: Thu, 1 Aug 2002 11:51:10 -0500

Jim,

Thanks for your comments.  I've delayed replying to your email in the hopes
that this would generate some additional discussion, but since there has
been no responses, and the deadline for feedback on the document is tomorrow
(Friday, August 2nd), I thought I would respond with a somewhat constructive
suggestion on how to allay some of your concerns. I have some specific
responses that I've embedded below, as indicated by [MB], as well as a
general proposal to add some specific text to the Overview section
describing the intent of Section 3 as follows:

Regards,
Mary H. Barnes
[email-masked]
972-684-5432
Wireless 817-703-4806

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, July 31, 2002 2:36 PM
To: midcom
Subject: [midcom] Comments on protocol evaluation draft

Middlebox experts,

Overall, I think this work is great. Mary Barnes should be complimented
and thanked for pulling this together.

While I have a few small issues with the facts or claims in Sections
1 and 2, my real concern is with Section 3. The concern is not so much
with the facts presented there, but with the way they might be
interpreted *as presented there*.

What the MIDCOM WG has tried to do in this protocol evaluation, and I
think largely succeeded in doing, is to compare the frameworks and
capabilities of a number of protocols to the MIDCOM requirements and
framework (Many of the MIDCOM requirements addressed the matching of
MIDCOM and protocol framework elements.). If you look at this on a MIDCOM
requirement by requirement (and / or framework element by framework
element basis) against the framework elements and protocol capabilities
of a single protocol, there are a number of possibilities:

- The protocol's framework elements and protocol capabilities match
exactly (or extremely close to exactly) the MIDCOM requirement or
framework element. GREAT! This is what we were looking and hoping for.

- The protocol does not have any framework elements or protocol
capabilities that address the MIDCOM requirement or framework element.
Not so great, but, again, this is what we were looking (but not hoping)
for.

- The protocol's framework elements and protocol capabilities somewhat
match the MIDCOM requirement or framework element. Here's where I start
to have some problems.

Where the match is not exact, there's two possibilities (or a
combination of them):

- In the individual protocol evaluation, a proposal is made to modify
or extend the protocol's mandatory framework elements or protocol
capabilities, or to modify optional framework elements or protocol
capabilities, to match the MIDCOM requirement or framework element.
From our perspective, this is good, but it may not be so good to other
users of the protocol. If the protocol is single purpose with few users
at this time and no guarantee of forward compatibility (e.g., RSIP),
modifying the protocol ain't so bad. But if the protocol has other
purposes, many users, or the guarantee of forward compatibility, this
may not be so good (The implications of "other purposes" and "many
users" is hopefully obvious. I'll return to the issue of guaranteed
forward compatibility below.). Also, extending the protocol with
optional framework elements and capabilities is probably not bad.

[MB] My view would be that P+ and P should be used to distinguish between
the things that could impact other users of the protocol. For example,
extensions that are inline with the general intent of the original protocol,
that don't cause any compatibility problems with existing uses of that
protocol should be P+.  However, if the extensions would impact existing
users (i.e. the changes required for MIDCOM also require changes to other
users of the protocol), then it should only be a P at the most. 
[/MB]

- Alternately or in combination, a proposal is made to "think about the
MIDCOM requirement or framework element in a different way" so that the
match is exact or almost so. I think there's quite a bit of this going
on in the protocol evaluations. Taken individually, if the proposed
change in thinking is not large, this might not be bad. Taken together,
if the sum of all such changed thinking for a protocol is not large,
this may also be not bad.

[MB] My view would be that at most these cases can only be P at most, and
perhaps some of them should be Fs.  I do agree that some of the evaluations
erred on the liberal side.  
[/MB]

But, if taken together the sum of all such proposals for a given
protocol is large, then essentially this says that the MIDCOM framework
or requirements would need some rather extensive revision to match the
protocol's framework and capabilities. That's probably bad.
[MB] I still think that the categories of P+ versus P helps from this
perspective. [/MB] 

For both of the above alternatives, the protocol would have been scored
as partially matching the MIDCOM requirements. But the distinction
between the two alternatives is lost in the scoring, so that the count of
the partial matches can not be easily evaluated as being overall good or
overall bad.

[MB] Per my comments above, I think the intent was for there to be a clear
distinction between P and P+ for this purpose.  Can you point out some of
the specific ones for which you see this problem?  I believe that we should
all be in agreement on the P+'s.  Whether some of the Ps should really be Fs
is another point for discussion. 
[/MB]

I didn't go through the exercise (and maybe I should) of marking each
partial requirements match as one or the other of the above and
counting the number of each alternative for each protocol, but my gut
feeling is that we may have a problem here.
[MB] Please do go through this exercise. P+ should all clearly be things
that don't break other users.
[/MB]

Returning to the issue of guaranteed forward compatibility, if the
proposal is to modify or extend mandatory framework elements or protocol
capabilities, or to modify optional elements or capabilities, then all
other users of the protocol, or its modified optional elements and
capabilities, would have to support the changes that were made to the
protocol for MIDCOM. Depending on the size of these changes, this may
not be acceptable to the protocols other users. And we probably
shouldn't consider these types of changes without those users buying
into them. Again, I didn't specifically count the applicable scorings
here and maybe I should have, but in this case I don't really think we
have much of a problem.

Another issue that I have with the whole evaluation is that it did not
consider the impact of a protocol's mandatory framework elements and
protocol capabilities that are *not* required by MIDCOM. The problem
here is that middleboxes and agents / applications may have to support
these even though they're not used. If the protocol has another
simultaneous use in the middlebox or agent / application, then the
impact is not so bad, in fact there may be no impact at all. But if
the use of the protocol is its first use in the middlebox or agent /
application, then I think we have to seriously consider this potential
problem. This problem relates to, but is not exclusive to, library and
source code reuse, which in general is desirable. But if the library or
source code has a lot of unused capabilities, and the MIDCOM implementer
can't (afford to) remove them, then this will place an unnecessary cost
on implementations.

[MB] I do think that the Diameter evaluation did bring forth this issue, but
certainly there was no specific requirement, so it's not captured in the
T/P+/P/F totals. However, it is brought out in the textual summary.  I do
think that some of this is brought out with the proposed changes I posted to
the list for the SNMP, MEGACO and COPS descriptions in section 1.
[/MB] 

I have no idea how to evaluate this in advance, but I have a gut feeling
we may have a *real* problem here. I am certainly open to ideas on how
to evaluate this in advance. But maybe we'll just have to wait for
specific protocol proposals to be able to see through this issue.

[MB] I don't think your alone in having some concerns about the process that
the WG is using, however, per my proposal above, this document is just one
piece of information that goes into the overall Protocol decision.  This
needs to be considered along with the semantic analysis which is underway.
We had this discussion right after IETF-53, and the decision was that this
document could be worked in parallel with the semantic analysis, but I think
there was general agreement that the semantic analysis was an equally (and
for some folks) more important part of the overall protocol decision
process. 
[/MB]

So, overall, I think the evaluation is good, but I have some issues with
how the facts presented, especially the summary counts, may be
interpreted.

[MB] Per my comments above, the document should be explicit about P+ versus
P any specific concerns should be raised.  As a minimum, I'm proposing to
add the text described above to ensure that the intent of Section 3 is
clear. 

Comments expected and welcome.

Jim

_______________________________________________
midcom mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/midcom

---

From: "Mary Barnes" <[email-masked]>
Date: Thu, 1 Aug 2002 11:59:41 -0500

Hi all,

Sorry, I hit SEND too soon, before I had added the proposed text (and
converted to Plain text):

"Section 3 provides a summary of the evaluation.  A numerical breakdown of
the applicability of the requirements, to the each of the protocols, for the
following categories is provided: Fully met, Partially met through the use
of extensions, Partially met through other changes to the protocol, or
Failing to be met.  This summary is not meant to provide a conclusive
statement of the suitability of the protocols, but rather to provide
information to be considered as input to the protocol decision process." 

Mary. 

-----Original Message-----
From: Barnes, Mary [NGC:B602:EXCH] 
Sent: Thursday, August 01, 2002 11:51 AM
To: [email-masked]'; midcom
Subject: RE: [midcom] Comments on protocol evaluation draft

Jim,

Regards,
Mary H. Barnes

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, July 31, 2002 2:36 PM
To: midcom
Subject: [midcom] Comments on protocol evaluation draft

Middlebox experts,

Jim

---

From: [email-masked]
Date: Thu, 1 Aug 2002 23:45:12 -0500

Mary (and other middlebox experts),

Thank you much for your reply.

The clarification of the grading system is especially useful. I had a
lot of trouble understanding it before, but I think I got it now.

Here's what I now understand the grades to mean:
T  - Compliance to the requirement with the protocol as is
P+ - Compliance to the requirement with an extension to the protocol
P  - Compliance to the requirement with a change, or an extension that
     impacts existing users of the protocol
F  - No compliance (even with extension of the protocol???)

Some observations on this:

- Since all the protocols are extensible and extensions can generally be
made, one way or another, to be backward compatible, I'm surprised that
we have any P's of F's. :-)

- We seem to have lost the concept of how large an extension needs to be
made to add a "thing" to the protocol. Is the protocol close to the
requirement and just needs a little extension? Or does the protocol have
nothing to offer and the thing has to be added completely? I originally
thought that was the idea behind the the T, P and F grades ("How much do
ya have to change the protocol to satify the requirement? None, a little,
and a lot, respectively"), but I guess we evolved from that, assuming
that was the original intention.

As you suggested, I went back and looked for what I thought were incon-
sistencies in P+ and P grades. Once I got started digging, what I wound
up doing was quickly reviewing all the scores for consistency with the
grading system as I now understand it and the summary information for
each protocol's compliance with each requirement as given in the
protocol evaluation I-D (BTW, I used the -02 version of the I-D, without
any of the changes that have been discussed since it was published.).

The grading inconsistencies that I saw fell into 4 categories:

1. The requirement compliance text for the protocol basically said "This
can be done with an extension to this protocol" and the grade was a T,
instead of a P+ as I believe it should be. I included the closely related
"The protocol does not preclude this extension from being made." in this
category.

2. The requirement compliance text for the protocol says the protocol
has something here but extension is required and can be made with no
impact, but graded it as a P instead of a P+.

3. The requirement compliance text for the protocol says the protocol has
nothing here but the necessary extension can be made without impact, and
graded it as an F instead of a P+.

4. The requirement compliance text for the protocol says the protocol has
nothing here but if ya use TLS or IPsec or whatever for security ya get
everything, and graded it as a T.

In the last case, I'm not sure what the right score is, but I would
suggest at most P+, to distinguish these from protocols that have the
right thing here without resorting to TLS or IPsec or whatever for
security, which I think should get a T grade. (For consistency, I
considered grades of F (Honestly given, in my opinion, but I could be
wrong.) but the requirement could be fulfilled with TLS or IPsec or
whatever for security, hence now graded P+ for consistency, as instances
of this case also.)

I also found a few other what I believe to be inconsistencies, at least
based on the scoring system as I now understand it. Call this category 5.

I list the inconsistencies I found at the end of this message, but
here's some statistics. Take 'em for whatever ya think they're worth.

Out of a total of 135 scores (27 requirements times 5 protocols) I found
what I thought were 31 inconsistencies. That's just less than 23%.

The number of inconsistencies by category by protocol are:

Category        SNMP    RSIP    MEGACO   DIAMETER        COPS    Total
--------        ----    ----    ------   --------        ----    -----
1        1       0       3       3        1       8
2        0       2       0       1        0       3
3        0       4       0       0        0       4
4        0       5       4       5        0      14
5        0       2       0       0        0       2
        --      --      --      --       --      --
Total    1      13       7       9        1      31

The resulting adjustments in protocols scores are:

Protocol        T       P+      P        F
--------        ---     ---     ---      ---
SNMP    - 1     + 1
RSIP    + 2     +10     - 3     - 9
MEGACO  - 7     + 7
DIAMETER        - 8     + 9     - 1
COPS    - 1     + 1
        ---     ---     ---     ---
Total   -15     +28     - 4     - 9

And the resulting protocol scores are:

Protocol        T       P+      P        F       Total
--------        ---     ---     ---      ---     -----
SNMP     18       9       0       0       27
RSIP     14      13       0       0       27
MEGACO   13      11       2       1       27
DIAMETER         13      13       1        0      27
COPS     19       6       1       1       27
        ---     ---     ---     ---      ---
Total    77      52       4       2      135

This was done real quick, so there's probably still stuff not right, but
now, at least to me, the numbers better match the general feeling I get
when reading the I-D.

Jim

================

Inconsistencies I found in the MIDCOM protocol evaluation grades
(This was composed with tab stops every 8 characters.)

                       Score
Require-                -------------------              Inconsistency
ment    Protocol        Was     Should be                Category
--------        --------        ---      ---------               --------
2.1.2   RSIP    P       T                5
2.1.8   RSIP    F       P+               4
        MEGACO  T       P+               4
        DIAMETER        T       P+               4
2.2.1   RSIP    P+      T                5
2.2.2   SNMP    T       P+               1
        RSIP    P       P+               2
        MEGACO  T       P+               1
        DIAMETER        T       P+               1
2.2.3   RSIP    P       P+               3
2.2.4   MEGACO  T       P+               1
        DIAMETER        T       P+               1
2,2,6   DIAMETER        T       P+               1
        COPS    T       P+               1
2.2.7   RSIP    F       P+               3
        DIAMETER        T       P+               1
2.2.8   RSIP    P       P+               2
        MEGACO  T       P+               1
2.2.9   RSIP    F       P+               3
2.2.11  RSIP    F       P+               3
2.3.1   RSIP    F       P+               4
        MEGACO  T       P+               4
        DIAMETER        T       P+               4
2.3.2   RSIP    F       P+               4
        MEGACO  T       P+               4
        DIAMETER        T       P+               4
2.3.3   RSIP    F       P+               4
        MEGACO  T       P+               4
        DIAMETER        T       P+               4
2.3.4   RSIP    F       P+               4
        DIAMETER        T       P+               4

---

From: "Tom-PT Taylor" <[email-masked]>
Date: Fri, 2 Aug 2002 10:00:47 -0400

From my point of view you raise two issues I cannot leave unanswered.  The
first has to do with P+ extensions and your first scoring inconsistency, the
second with security.

I confess that the need for protocol extensions poses a logical issue: how
to treat this need when it affects multiple categories of response.  On the
one hand, I think it is over-stating things to simply score 1 against each
category that depends on the extension.  On the other, it is certainly true
that, whether you are defining a new MIB, a new DIAMETER application, or a
new Megaco package, you imply the need for extra code in both the Agent and
the Middlebox above that supplied by the base protocol implementation.  I
myself would propose adding an evaluation category that specifies the
increment to the base protocol implied by its use for MIDCOM.  Leaving that
aside, any protocol that fails to meet one of the MIDCOM requirements is
probably beyond our consideration.

On the security issue: you dismiss too quickly those protocols that rely on
lower layers for their necessary degree of security.  The RFCs specifying
these protocols have security sections.  The protocol security architecture
in each case was arrived at by considering the specific threats that had to
be countered in the protocol's domain of application.  An architecture that
relies on lower layer security is perfectly valid if that meets the
requirements for that protocol.

What we do have to think about is whether the threats are different for the
MIDCOM application from those considered when the candidate protocol was
being designed.  If so, the security architecture for that protocol may have
to be modified to fit MIDCOM application.  But the security section of the
MIDCOM requirements document reflects the results of the MIDCOM threat
analysis.  Thus if our candidate protocols, including their specified
security architectures, can meet these requirements in a practical way, they
must be scored as passing those requirements.  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Friday, August 02, 2002 12:45 AM
To: Barnes, Mary [NGC:B602:EXCH]
Cc: 'midcom'
Subject: RE: [midcom] Comments on protocol evaluation draft

Protocol        T       P+      P        F
--------        ---     ---     ---      ---
SNMP    - 1     + 1
RSIP    + 2     +10     - 3     - 9
MEGACO  - 7     + 7
DIAMETER        - 8     + 9     - 1
COPS    - 1     + 1
        ---     ---     ---     ---
Total   -15     +28     - 4     - 9

Protocol        T       P+      P        F       Total

Jim

================

                       Score

---

From: "Mary Barnes" <[email-masked]>
Date: Fri, 2 Aug 2002 19:13:40 -0500

Hi Jim,

I did finally complete a more detailed review of your suggested changes, and
I think you've interpreted text, which I've understood to describe how the
requirement is met by the specific protocol,  to mean that extensions are
required to support that functionality and I don't think that's the case
for:
2.2.2 SNMP
2.2.4 Megaco
2.2.4 Diameter
2.2.6 Diameter
2.2.6 COPS 

I think this same applies to 2.2.7 for Diameter, but I think this needs to
be reworded as a positive statement, rather than it's current negative form.
I'll post the proposed text separately.  

I think you may have a valid point for 2.2.8 Megaco.  I'll start a separate
thread on that one. 

For 2.1.8, Diameter is currently a P and per my email last week, I will be
changing this to a P+, so we're also in agreement on what that one should
be. 

This along with my email response this morning should provide complete
feedback on how I propose to handle each of your suggested changes.

Thanks,
Mary. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, August 01, 2002 11:45 PM
To: Barnes, Mary [NGC:B602:EXCH]
Cc: 'midcom'
Subject: RE: [midcom] Comments on protocol evaluation draft

Protocol        T       P+      P        F
--------        ---     ---     ---      ---
SNMP    - 1     + 1
RSIP    + 2     +10     - 3     - 9
MEGACO  - 7     + 7
DIAMETER        - 8     + 9     - 1
COPS    - 1     + 1
        ---     ---     ---     ---
Total   -15     +28     - 4     - 9

Protocol        T       P+      P        F       Total

Jim

================

                       Score