Subject: [RTG-DIR] Please start using the Document QA process now

From: Alia Atlas <[email-masked]>
Date: Wed, 20 Aug 2014 10:45:33 -0400

Hi to all the Routing  Area WG Chairs and Secretaries,

As Adrian and I discussed prior to IETF 90 and in the Routing Area meeting
at IETF 90,
we would like to start using the new Draft Quality Assurance process (
http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa ) for all drafts
that are being
considered for WG adoption.   Drafts that are already WG drafts do not need
to go through this process - but chairs can ask if it seems desirable.

As with many processes in the IETF, this process is intended to be used for
the spirit of the goal, subject to common-sense and technical perspective.
In particular, none of this process is intended to gate or otherwise impede
the normal progression of documents through IETF working groups: rather, it
is intended to provide additional input, review, and comments that the
working groups can use to help improve the quality of their documents, and
to help chairs determine quality and consensus for documents. We do not
expect IETF process to wait for reviews to happen.

As described on the wiki page, the detailed process is:

   1. The WG Chair or Secretary emails the Routing Directorate Coordinators
   ( Deborah Brungard <db3546 at att.com>) and Jonathan Hardwick
   <jonathan.hardwick at metaswitch.com>). The subject should be WG Draft
   QA Requested: <draft-name <http://tools.ietf.org/html/draft-name>>.

   1. The WG Chair or Secretary marks a draft in the datatracker as
   "Candidate for Adoption" and specifies the date on which WG Draft QA was
   requested.

   1. The Routing Directorate Coordinators and WG Chairs agree and find a
   QA Reviewer. The review can be done during the WG Call for Adoption. The WG
   Chair or Secretary adds a comment in the draft's datatracker history page
   indicating whom is the QA Reviewer.

Please start doing this now.

After IETF-91, we can discuss how this is going and its usefulness.

Thanks,
Alia

---

From: Loa Andersson <[email-masked]>
Date: Wed, 20 Aug 2014 18:03:32 +0200

Alia

A few quick questions on this. In the message that the RTG ADs sent
to the wg chairs and directorate when the discussion on re-org started
you said

"First, we intend to use our Routing Directorate more proactively by
  introducing a Working Group Draft Quality Assurance (WG Draft QA)
  process where the same selected routing directorate member will review
  a draft during WG draft adoption and during WG last call.  The process
  will be documented on the Routing Area wiki.  This should allow
  directorate reviews to report technical issues that can actually get
  fixed early in the process (equiv of bug reports) as opposed to just
  noting the concerns in the drafts (equiv of release notes)."

It seems that there are a changes from what was said at that time
and what is said now.

"... where the same selected routing directorate member will review
  a draft during WG draft adoption and during WG last call"

while we now say

"...for all drafts that are being considered for WG adoption."

What motivated this change?

Second, some wg's have review teams that does something similar to
the WG Draft QA process just prior to that the the call for working
group adoption is started.

Do we do the  WG Draft QA in addition to, instead of these reviews,
or can we continue doing the review team reviews instead of WG Draft QA?

/Loa

On 2014-08-20 16:45, Alia Atlas wrote:
> As Adrian and I discussed prior to IETF 90 and in the Routing Area
> meeting at IETF 90,
> http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa ) for all
> drafts that are being
> considered for WG adoption.   Drafts that are already WG drafts do not
> need to go through this process - but chairs can ask if it seems desirable.
>
> As with many processes in the IETF, this process is intended to be used
> for the spirit of the goal, subject to common-sense and technical
> perspective. In particular, none of this process is intended to gate or
> otherwise impede the normal progression of documents through IETF
> working groups: rather, it is intended to provide additional input,
> review, and comments that the working groups can use to help improve the
> quality of their documents, and to help chairs determine quality and
> consensus for documents. We do not expect IETF process to wait for
> reviews to happen.
>
>  1. The WG Chair or Secretary emails the Routing Directorate
>     Coordinators ( Deborah Brungard <db3546 at att.com
>     <http://att.com/>>) and Jonathan Hardwick <jonathan.hardwick at
>     metaswitch.com <http://metaswitch.com/>>). The subject should be WG
>     Draft QA Requested: <draft-name
>     <http://tools.ietf.org/html/draft-name>>.
>
>  2. The WG Chair or Secretary marks a draft in the datatracker as
>     "Candidate for Adoption" and specifies the date on which WG Draft QA
>     was requested.
>
>  3. The Routing Directorate Coordinators and WG Chairs agree and find a
>     QA Reviewer. The review can be done during the WG Call for Adoption.
>     The WG Chair or Secretary adds a comment in the draft's datatracker
>     history page indicating whom is the QA Reviewer.
>

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert                          [email-masked]
Huawei Technologies (consultant)     phone: +46 739 81 21 64

---

From: Alia Atlas <[email-masked]>
Date: Wed, 20 Aug 2014 12:32:03 -0400

Hi Loa,

> What motivated this change?

This isn't intended to be a change.  A draft that is going to have a WG call
for adoption issued should get a QA reviewer so that the review can happen
ideally before the WG call for adoption ends.

Basically, I see  the process as:

1. Individual draft submitted and discussed.
2. WG Chairs decide to issue a call for adoption.
2.1 WG Chairs get a QA reviewer with review
2.2 WG Chairs issue a call for adoption (can be before QA reviewer is
assigned if
that is taking too much time)
2.3 Ideally, QA reviewer contributes the review to the WG mailing list
2.4 WG Chairs decide if there is active consensus to adopt the draft.
3. Draft is improved based upon reviews and comments.

Where MPLS has additional process between considering a draft and doing a
WG
call for adoption that is treating those early reviews as gating, I can
understand
that you may see a difference between the two states.

> group adoption is started.

Right - that is intended really for review inside the WG and before the
call for
WG adoption is started.

The WG Draft QA is intended more for cross-WG review and expected to
take place in parallel with the call for adoption.

> or can we continue doing the review team reviews instead of WG Draft QA?

I believe that they are addressing slightly different problems.  For MPLS,
I think
that some of the drafts could benefit from early review that considers
cross-WG
concerns as well as usefulness of the draft.

That said, the process does allow for WG chairs to decide not to request a
QA
reviewer or to request one who isn't part of the routing directorate.  We
just ask
that it's documented so that we can tell the difference between a decision
not to
request and a failure to remember to do so.

I'd prefer to see the QA reviews done so we can determine how much value is
being added rather than simply say it's not needed for a particular WG.

Regards,
Alia

/Loa

---

From: Loa Andersson <[email-masked]>
Date: Wed, 20 Aug 2014 19:42:17 +0200

Alia,

I think you missed the point. Initially we had two QA reviews intended. One at wg adoption poll and a second at wglc. The directive going out now only talks about a RTG dir review at adoption poll intended or not it is a change. Why did we change. 

/Loa

Sent from my iPhone

> On 20 aug 2014, at 18:32, Alia Atlas <[email-masked]> wrote:
> 
> Hi Loa,
> 
> 2.2 WG Chairs issue a call for adoption (can be before QA reviewer is assigned if
> Where MPLS has additional process between considering a draft and doing a WG 
> call for adoption that is treating those early reviews as gating, I can understand
> Right - that is intended really for review inside the WG and before the call for
> WG adoption is started.
> 
> I believe that they are addressing slightly different problems.  For MPLS, I think
> that some of the drafts could benefit from early review that considers cross-WG 
> That said, the process does allow for WG chairs to decide not to request a QA 
> reviewer or to request one who isn't part of the routing directorate.  We just ask 
> that it's documented so that we can tell the difference between a decision not to

---

From: Alia Atlas <[email-masked]>
Date: Wed, 20 Aug 2014 14:37:12 -0400

Loa,

On Wed, Aug 20, 2014 at 1:42 PM, Loa Andersson <[email-masked]> wrote:

> Alia,
>
> I think you missed the point. Initially we had two QA reviews intended.
> One at wg adoption poll and a second at wglc. The directive going out now
> only talks about a RTG dir review at adoption poll intended or not it is a
> change. Why did we change.
>

Oh - there isn't a change in terms of the QA reviews intended.  At the WG
adoption poll is when the QA reviewer is assigned - and this is talking
about that process.  Once the QA reviewer is assigned, he or she is
expected to do both of the reviews.   Please do read the wiki and
make/suggest changes for clarification if necessary.

Regards,
Alia

---

From: Ross Callon <[email-masked]>
Date: Wed, 20 Aug 2014 20:09:59 +0000

I am wondering how this fits with WG review teams, where there are such.

As an example, the MPLS WG has its own review team. We just finished the review team review of draft-akiya-mpls-lsp-ping-reply-mode-simple (with reviews from all 3 reviewers who were asked). The document was updated accordingly.

Do we want a document to go through WG specific review, then routing area QA team review, then call for adoption, and later go through last call, WG chair review, QA team review (again), AD review, IETF last call, and IESG review?

An alternative would be to replace one of the WG review team reviewers with one or two routing area QA reviewers, and do these two reviews at the same time. At least in the MPLS WG we typically do the MPLS review team reviews before the call for WG adoption.

Thanks, Ross

From: rtg-dir [mailto:[email-masked]] On Behalf Of Alia Atlas
Sent: Wednesday, August 20, 2014 10:46 AM
To: [email-masked]; [email-masked]
Subject: [RTG-DIR] Please start using the Document QA process now

As Adrian and I discussed prior to IETF 90 and in the Routing Area meeting at IETF 90,
we would like to start using the new Draft Quality Assurance process ( http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa ) for all drafts that are being
considered for WG adoption.   Drafts that are already WG drafts do not need to go through this process - but chairs can ask if it seems desirable.

As with many processes in the IETF, this process is intended to be used for the spirit of the goal, subject to common-sense and technical perspective. In particular, none of this process is intended to gate or otherwise impede the normal progression of documents through IETF working groups: rather, it is intended to provide additional input, review, and comments that the working groups can use to help improve the quality of their documents, and to help chairs determine quality and consensus for documents. We do not expect IETF process to wait for reviews to happen.

As described on the wiki page, the detailed process is:
1.      The WG Chair or Secretary emails the Routing Directorate Coordinators ( Deborah Brungard <db3546 at att.com<http://att.com/>>) and Jonathan Hardwick <jonathan.hardwick at metaswitch.com<http://metaswitch.com/>>). The subject should be WG Draft QA Requested: <draft-name<http://tools.ietf.org/html/draft-name>>.
2.      The WG Chair or Secretary marks a draft in the datatracker as "Candidate for Adoption" and specifies the date on which WG Draft QA was requested.
3.      The Routing Directorate Coordinators and WG Chairs agree and find a QA Reviewer. The review can be done during the WG Call for Adoption. The WG Chair or Secretary adds a comment in the draft's datatracker history page indicating whom is the QA Reviewer.
Please start doing this now.

Thanks,
Alia

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 21 Aug 2014 05:13:25 +0000

Ross, Alia and all,

Apologies for probably speaking out of turn, but just for my education: Could somebody please tell me which Routing Area WGs beside MPLS, have their own review teams?

The context for my question:

  *   As of today, there are 20 WGs in the Routing Area (the number seems to remain the same after restructuring). I try to follow 9 of them, and MPLS is one of these I follow
  *   MPLS-RT is frequently mentioned on the mailing list. To me, this is a sign of its being very much alive and kicking
  *   I do not remember review teams being mentioned on the mailing lists of the other 8 WGs. Could be my omission/senility, of course, but could also mean that they either do not exist or are not very active.
  *   It could be nice for me to know which case it is (could be a mix, of course:-)...

Regards, and lots of thanks in advance,

Sasha

________________________________
From: rtg-dir <[email-masked]> on behalf of Ross Callon <[email-masked]>
Sent: Wednesday, August 20, 2014 11:09 PM
To: Alia Atlas; [email-masked]; [email-masked]
Subject: Re: [RTG-DIR] Please start using the Document QA process now

Thanks, Ross

Thanks,
Alia

---

From: Loa Andersson <[email-masked]>
Date: Thu, 21 Aug 2014 09:43:39 +0200

Ross,

I believe that the Document QA process looks like this, though it could
be clearer.

MPLS-RT review
WG Adoption Poll/RTG Dir review
    Note: The RTG Dir reviewer is supposed to follow the document through
    the period from Adoption Poll to WGLC.
WGLC/RTG Dir review

I.e. the RTG Dir review is "part of" the adoption poll and wglc.

The MPLS-RT is (so far) MPLS wg specific.

/Loa

On 2014-08-20 22:09, Ross Callon wrote:
> As an example, the MPLS WG has its own review team. We just finished the
> review team review of draft-akiya-mpls-lsp-ping-reply-mode-simple (with
> reviews from all 3 reviewers who were asked). The document was updated
> accordingly.
>
> Do we want a document to go through WG specific review, then routing
> area QA team review, then call for adoption, and later go through last
> call, WG chair review, QA team review (again), AD review, IETF last
> call, and IESG review?
>
> An alternative would be to replace one of the WG review team reviewers
> with one or two routing area QA reviewers, and do these two reviews at
> the same time. At least in the MPLS WG we typically do the MPLS review
> team reviews before the call for WG adoption.
>
> Thanks, Ross
>
> *From:*rtg-dir [mailto:[email-masked]] *On Behalf Of *Alia Atlas
> *Sent:* Wednesday, August 20, 2014 10:46 AM
> *To:* [email-masked]; [email-masked]
> *Subject:* [RTG-DIR] Please start using the Document QA process now
>
> 1.The WG Chair or Secretary emails the Routing Directorate Coordinators
> ( Deborah Brungard <db3546 at att.com <http://att.com/>>) and Jonathan
> Hardwick <jonathan.hardwick at metaswitch.com
> <http://metaswitch.com/>>). The subject should be WG Draft QA Requested:
> <draft-name <http://tools.ietf.org/html/draft-name>>.
>
> 2.The WG Chair or Secretary marks a draft in the datatracker as
> 3.The Routing Directorate Coordinators and WG Chairs agree and find a QA
> Reviewer. The review can be done during the WG Call for Adoption. The WG
> Chair or Secretary adds a comment in the draft's datatracker history
> page indicating whom is the QA Reviewer.
>

--

---

From: Loa Andersson <[email-masked]>
Date: Thu, 21 Aug 2014 09:54:27 +0200

Alia,

To put it simply!

As for the documents that we start wg adoption poll for we also request
RTG Dir review, and that reviewer will stay with the document through
the wg process and do a new RTG Dir review at wglc. Right?

As for documents that we already have a wg documents, we won't request
RTG Dir reviews, unless we want them for some other reason than
Document QA. Right?

/Loa

On 2014-08-20 20:37, Alia Atlas wrote:
> Loa,
>
> On Wed, Aug 20, 2014 at 1:42 PM, Loa Andersson <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Alia,
>
>     I think you missed the point. Initially we had two QA reviews
>     intended. One at wg adoption poll and a second at wglc. The
>     directive going out now only talks about a RTG dir review at
>     adoption poll intended or not it is a change. Why did we change.
>
>
> Oh - there isn't a change in terms of the QA reviews intended.  At the
> WG adoption poll is when the QA reviewer is assigned - and this is
> talking about that process.  Once the QA reviewer is assigned, he or she
> is expected to do both of the reviews.   Please do read the wiki and
>     On 20 aug 2014, at 18:32, Alia Atlas <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>>     Hi Loa,
>>
>>     On Wed, Aug 20, 2014 at 12:03 PM, Loa Andersson <[email-masked]
>>     <mailto:[email-masked]>> wrote:
>>
>>         Alia
>>
>>             http://wiki.tools.ietf.org/__area/rtg/trac/wiki/RtgDirDocQa <http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa>
>>             ) for all
>>             drafts that are being
>>             <http://att.com>
>>             metaswitch.com <http://metaswitch.com>
>>             <http://metaswitch.com/>>). The subject should be WG
>>                 <http://tools.ietf.org/html/__draft-name
>>             <http://tools.ietf.org/html/draft-name>>>.
>>
>>         Loa Andersson                        email:
>>         [email-masked] <mailto:[email-masked]>
>>         Senior MPLS Expert [email-masked] <mailto:[email-masked]>
>>         <tel:%2B46%20739%2081%2021%2064>
>>
>>
>

--

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Aug 2014 10:06:48 -0400

Loa,

Yes, exactly.

Alia

On Thu, Aug 21, 2014 at 3:54 AM, Loa Andersson <[email-masked]> wrote:

>>>             http://wiki.tools.ietf.org/__area/rtg/trac/wiki/RtgDirDocQa
>>> <http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa>
>>>
>>>             ) for all
>>>             drafts that are being

---

From: Lou Berger <[email-masked]>
Date: Thu, 21 Aug 2014 10:11:22 -0400

Alia,
    I'm unclear on one point: Is it your intent for the quality review
to gate WG acceptance?

Lou

On 8/20/2014 10:45 AM, Alia Atlas wrote:
> we would like to start using the new Draft Quality Assurance process
> ( http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa ) for all
> drafts that are being
> As with many processes in the IETF, this process is intended to be
> used for the spirit of the goal, subject to common-sense and technical
> perspective. In particular, none of this process is intended to gate
> or otherwise impede the normal progression of documents through IETF
> review, and comments that the working groups can use to help improve
> the quality of their documents, and to help chairs determine quality
> and consensus for documents. We do not expect IETF process to wait for
> reviews to happen.
>
>     <http://att.com/>>) and Jonathan Hardwick <jonathan.hardwick
>     at metaswitch.com <http://metaswitch.com/>>). The subject should
>     be WG Draft QA Requested: <draft-name
>     "Candidate for Adoption" and specifies the date on which WG Draft
>     QA was requested.
>
>  3. The Routing Directorate Coordinators and WG Chairs agree and find
>     a QA Reviewer. The review can be done during the WG Call for
>     Adoption. The WG Chair or Secretary adds a comment in the draft's
>     datatracker history page indicating whom is the QA Reviewer.
>

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Aug 2014 10:52:16 -0400

Sandy,

Exactly so.    :-)

Alia

On Thu, Aug 21, 2014 at 10:50 AM, Sandra Murphy <[email-masked]> wrote:

> I read through the process again and to me it seems clear that it does not.
>
> "Any issues the QA Reviewer finds are written down, sent to the mailing
> list and discussed for future versions, but need not gate the progress of
> the draft in the WG"
>
> I'd be interested to see how others interpret the process.
>
> --Sandy
>
> On Aug 21, 2014, at 10:11 AM, Lou Berger <[email-masked]> wrote:
>
> > Alia,

---

From: Lou Berger <[email-masked]>
Date: Thu, 21 Aug 2014 11:12:14 -0400

Okay, then why not wait for assignment until there is a WG -00 draft? 
This way we avoid wasting reviewer resources on rejected documents.

Also, wither way the next step should be added as a 4. to the "detailed
process".

Thanks,
Lou
On 8/21/2014 10:52 AM, Alia Atlas wrote:
> Sandy,
>
> Exactly so.    :-)
>
> Alia
>
>
> On Thu, Aug 21, 2014 at 10:50 AM, Sandra Murphy <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     "Any issues the QA Reviewer finds are written down, sent to the
>     mailing list and discussed for future versions, but need not gate
>     the progress of the draft in the WG"
>
>     On Aug 21, 2014, at 10:11 AM, Lou Berger <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>     > Alia,
>     >>    at metaswitch.com <http://metaswitch.com>
>     <http://metaswitch.com/>>). The subject should

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Thu, 21 Aug 2014 15:17:13 +0000

I agree. 
Acee

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Aug 2014 11:22:00 -0400

Lou,

> This way we avoid wasting reviewer resources on rejected documents.

The QA reviewer's review isn't given special consideration to block WG
adoption
BUT it is intended to be useful input to the WG decision about whether to
adopt
the WG-00 draft.

To be blunt, it can be another tool to demonstrate lack of consensus or
that a draft
is a bad idea before the draft is actually adopted by the WG.

As for avoiding wasting reviewer resources, the gate there is that the WG
chairs
believe the draft should be up for WG adoption.   I would hope that is a
sufficient gate.

> Also, wither way the next step should be added as a 4. to the "detailed

Can you clarify?  I.e. suggest words  - and it is a wiki - feel free to add
a bit of text
with context  such as [NOTE: 4. blah blah (suggested by Lou Berger)]

Thanks,
Alia

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 21 Aug 2014 11:33:56 -0400

On Thu, Aug 21, 2014 at 11:29:08AM -0400, Christian Hopps wrote:
> Yeah, I thought it was an odd ordering as well.

There are competing goals here and I think that's part of the problem.

1. We want a document that "is a WG document" to clear quality issues as soon
as possible.  See the justifications in the wiki.

2. Documents that are of interesting technical content but have other issues
are difficult to read and review and those issues serve as a impediment to
discussion of the draft pre-adoption-call.

I believe we are talking about solving the first issue.

-- Jeff

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 21 Aug 2014 15:39:14 +0000

Jeffrey,
A naïve question: 
If a document is "difficult to read and review" (no matter at what stage), how can the WG understand/reach consensus  that it is  "of interesting technical content "? 

Or did I misread something in your mail?

Regards,
       Sasha 
Email: [email-masked]
Mobile: 054-9266302

> -----Original Message-----
> From: rtg-dir [mailto:[email-masked]] On Behalf Of Jeffrey Haas
> Sent: Thursday, August 21, 2014 6:34 PM
> To: Christian Hopps
> Cc: Sandra Murphy; [email-masked]; Lou Berger; [email-masked]; Alia
> Atlas
> 1. We want a document that "is a WG document" to clear quality issues as
> soon as possible.  See the justifications in the wiki.
>

---

From: Ross Callon <[email-masked]>
Date: Thu, 21 Aug 2014 15:53:57 +0000

Two points:

Yes, if a document is difficult to read and review, then we should expect that the WG will either have trouble coming to consensus that it is of interesting and valuable technical content, or should decide that it is not ready to be adopted. This is part of the reason that the MPLS WG has its review team look at documents prior to the call for adoption. 

I have come up with a different way to look at the routing area QA review: Looking at the IETF process as it has existed for many years, when a document is called for WG adoption, we as co-chairs might *hope* that multiple experienced WG participants who have a clue about the technology will read the document, evaluate whether it makes sense, and send their opinions, comments, and observations to the WG email list. This might in some cases be short and sweet ("IMHO the document is ready to be adopted as a WG item" or "IMHO this document is a really bad idea"). This might in some cases include detailed comments that should be used to update the document, either before or after adoption. Perhaps the routing area QA review might end up being little more than a way to get at least one experienced knowledgeable person to promise that they will indeed do precisely this review during the call for adoption. If we think of the review this way, then it does not actually add delay or more process to the existing IETF process, but it just makes it more likely that for each document at least some portion of our experts will actually do what we already want them to do. 

Ross

-----Original Message-----
From: rtg-dir [mailto:[email-masked]] On Behalf Of Alexander Vainshtein
Sent: Thursday, August 21, 2014 11:39 AM
To: Jeffrey Haas
Cc: [email-masked]; Christian Hopps; Sandra Murphy; Alia Atlas; [email-masked]; Lou Berger
Subject: Re: [RTG-DIR] Please start using the Document QA process now

Jeffrey,
A naïve question: 

Regards,
       Sasha

---

From: Sandra Murphy <[email-masked]>
Date: Thu, 21 Aug 2014 10:50:26 -0400

"Any issues the QA Reviewer finds are written down, sent to the mailing list and discussed for future versions, but need not gate the progress of the draft in the WG"

--Sandy

---

From: Christian Hopps <[email-masked]>
Date: Thu, 21 Aug 2014 11:29:08 -0400

Thanks,
Chris.

---

From: Christian Hopps <[email-masked]>
Date: Thu, 21 Aug 2014 11:37:31 -0400

On Aug 21, 2014, at 11:22 AM, Alia Atlas <[email-masked]> wrote:

> The QA reviewer's review isn't given special consideration to block WG adoption
> BUT it is intended to be useful input to the WG decision about whether to adopt
> the WG-00 draft.

Ok, I could see having an extra external voice being useful as a way of including extra support or non-support for a document if there was some contention.

Thanks,
Chris.

> To be blunt, it can be another tool to demonstrate lack of consensus or that a draft
> As for avoiding wasting reviewer resources, the gate there is that the WG chairs
> believe the draft should be up for WG adoption.   I would hope that is a sufficient gate.
>  
> Can you clarify?  I.e. suggest words  - and it is a wiki - feel free to add a bit of text

---

From: Lou Berger <[email-masked]>
Date: Thu, 21 Aug 2014 12:16:50 -0400

Alia,

On 8/21/2014 11:22 AM, Alia Atlas wrote:
> Lou,
>
> On Thu, Aug 21, 2014 at 11:12 AM, Lou Berger <[email-masked]
The following really confuses me:

> To be blunt, it can be another tool to demonstrate lack of consensus

Is this the point of the existing WG adoption poll?  How does an outside
review aid in a show of consensus by the WG?

> or that a draft
Isn't the the job of the chair to cover this case?  BTW Note I'm not
saying that they block the poll.  In the past I  have used polls on
documents in order to *stop* discussion on the topic.

> believe the draft should be up for WG adoption.   I would hope that is

This seems to completely counter your previous comment. If the chair can
do the job of gating the document here, they can do it for the previous
case to.

> Can you clarify?  I.e. suggest words  - and it is a wiki - feel free
So anyone can change your process???

How about something like:

4. The QA review does not block or gate a call for adoption, and the QA
Reviewer may send their comments to the WG and authors during the call,
or after it is closed. If the document is adopted, the QA Reviewer will
work with the document editors to resolve comments. Comments may be
tracked via the WG's trac tools page.

> with context  such as

> [NOTE: 4. blah blah (suggested by Lou Berger)]
>
Please no.  --  (or perhaps you're just trying to get me *not* to write
anything;-)

Lou

>     > On Thu, Aug 21, 2014 at 10:50 AM, Sandra Murphy
>     <[email-masked] <mailto:[email-masked]>
>     > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >
>     >     I read through the process again and to me it seems clear
>     that it
>     >     does not.
>     >
>     <mailto:[email-masked]>
>     >     <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >
>     >     > Alia,
>     >     >> considered for WG adoption.   Drafts that are already WG
>     drafts
>     >     do not
>     >     >> need to go through this process - but chairs can ask if
>     it seems
>     >     >> desirable.
>     >     >>
>     >     >> perspective. In particular, none of this process is
>     intended to
>     >     gate
>     >     >> or otherwise impede the normal progression of documents
>     through
>     >     IETF
>     <http://metaswitch.com>
>     >     >>    "Candidate for Adoption" and specifies the date on
>     which WG
>     >     Draft
>     >     >>    QA was requested.
>     >     >>
>     >     >> 3. The Routing Directorate Coordinators and WG Chairs
>     agree and
>     >     find

---

From: Lou Berger <[email-masked]>
Date: Thu, 21 Aug 2014 12:20:30 -0400

On 8/21/2014 11:53 AM, Ross Callon wrote:
>  If we think of the review this way, then it does not actually add delay or more process to the existing IETF process, but it just makes it more likely that for each document at least some portion of our experts will actually do what we already want them to do. 
Sounds like the reviewer really needs to be active in the WG then and
not just reviewing for document quality...

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 21 Aug 2014 12:20:35 -0400

Sasha,

On Thu, Aug 21, 2014 at 03:39:14PM +0000, Alexander Vainshtein wrote:
> A na?ve question: 

It seems to happen. Some people seem willing to push through difficult
language or confusing syntax anyway.  

Improving the quality of the document *before* the call to adopt means the
document can get better discussion and easier review during adoption.

Once a document has been adopted, the audience is wider.

-- Jeff

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Aug 2014 12:27:55 -0400

Lou,

> review aid in a show of consensus by the WG?

Good point - by triggering discussion?

> documents in order to *stop* discussion on the topic.

The primary purpose of the QA Review process is to get early cross-WG
review and routing directorate reviews done while the drafts are still
malleable.

> do the job of gating the document here, they can do it for the previous

If the WG Chair doesn't want a QA Reviewer for a particular draft, that is
up to the
WG chair's discretion.

> tracked via the WG's trac tools page.

Sounds good.  I think it is implied.

> Please no.  --  (or perhaps you're just trying to get me *not* to write

Without context after this discussion is fine too - but I beat you to it.

Alia

> >     >     >    I'm unclear on one point: Is it your intent for the
> quality
> >     >     review
> >     >     > to gate WG acceptance?
> >     >     >
> >     >     > Lou
> >     >     >
> >     >     >> ( http://wiki.tools.ietf.org/area/rtg/trac/wiki/RtgDirDocQa
> )
> >     >     for all
> >     >     >> drafts that are being
> >     >     >> and consensus for documents. We do not expect IETF process
> to
> >     >     wait for
> >     >     >> reviews to happen.
> >     >     >>

---

From: Alia Atlas <[email-masked]>
Date: Thu, 21 Aug 2014 12:32:25 -0400

On Thu, Aug 21, 2014 at 12:20 PM, Lou Berger <[email-masked]> wrote:

> >  If we think of the review this way, then it does not actually add delay
> or more process to the existing IETF process, but it just makes it more
> likely that for each document at least some portion of our experts will
> actually do what we already want them to do.

While it is desirable to have active WG participants review a draft, that
"should" happen because the draft is of interest  and relevant.   The WG
Document QA process is NOT trying to fix that problem.

The WG Document QA is trying to get the cross-WG perspective via early
directorate review when issues can still be easily fixed.

Of course, more reviews by experienced participants is desirable - but
that's part of having a WG with energy and drafts in the design center.
 Sometimes that comes by kicking (s/kicking/asking) people individually to
do reviews.

Alia

---

From: Lou Berger <[email-masked]>
Date: Thu, 21 Aug 2014 12:51:11 -0400

Alia,
    The updated page is helpful. The timing part of the thread is
covered below.

On 8/21/2014 12:27 PM, Alia Atlas wrote:
> Lou,
>
> On Thu, Aug 21, 2014 at 12:16 PM, Lou Berger <[email-masked]
>     > BUT it is intended to be useful input to the WG decision about
>     whether
>     > to adopt
>     > the WG-00 draft.
>     >

I think early QA review is a great idea, but I certainly hope you didn't
mean that once a document is accepted as a WG document it is no longer
"malleable".  WG change control of a document starts at adoption, not
before.

>  
>
>     > As for avoiding wasting reviewer resources, the gate there is
>     that the
>     > WG chairs
>     > believe the draft should be up for WG adoption.   I would hope
>     that is
>     > a sufficient gate.
>     >
>
> If the WG Chair doesn't want a QA Reviewer for a particular draft,
> that is up to the
> WG chair's discretion.
>

Giving the chair the discretion to submit a  QA Review request either
pre or post adoption addresses my concern.  (Looks like the process part
of the page will need some tweaking to make it clear that chairs have
this choice.)

Thanks!

...

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 21 Aug 2014 19:52:38 +0000

Jeff,
Lots of thanks for a prompt response.

I agree with you that some people are ready to push it through the text that is really difficult to read.
But this does not happen frequently IMO - and not because the majority of the drafts are well-written!

The result in many cases is that people simply do not care.

Having somebody assigned to read and review the document helps IMO. And sometimes his/her comments eventually result in a document that is much easier to read and understand, so that the WG consensus could be based on actual understanding what the draft is about and not just on default.
 
My 2c,
Sasha 

________________________________________
From: Jeffrey Haas <[email-masked]>
Sent: Thursday, August 21, 2014 7:20 PM
To: Alexander Vainshtein
Cc: Jeffrey Haas; Sandra Murphy; [email-masked]; Lou Berger; [email-masked]; Alia Atlas; Christian Hopps
Subject: Re: [RTG-DIR] Please start using the Document QA process now

Sasha,

-- Jeff

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 21 Aug 2014 16:03:52 -0400

Alia 

IDR has a draft that is in WG adoption call now, but early comments have caused the author to re-spin the draft.   

Three questions: 

1)      Should I wait until the authors re-spin the draft to ask for the reviewer or give a heads up to Deborah and Jonathan on the draft? 

2)      In my own personal reviewing of the draft, I have found existing drafts in L2VPN and IDR utilizing BGP solutions with 80-90% overlap.  I have also found an RFC approved by Softwires and not cross-reviewed in IDR.   Should I prepare a background scenario to go with the draft for the reviewer? 

3)      The authors of this draft have asked for early allocation.  Will the reviewer be ready to reply for the early allocation as well as WG Adoption and WG LC. 

Sue Hares

From: Alia Atlas [mailto:[email-masked]] 
Sent: Thursday, August 21, 2014 12:32 PM
To: Lou Berger
Cc: Ross Callon; [email-masked]; Christian Hopps; [email-masked]
Subject: Re: [RTG-DIR] Please start using the Document QA process now

While it is desirable to have active WG participants review a draft, that "should" happen because the draft is of interest  and relevant.   The WG Document QA process is NOT trying to fix that problem.

The WG Document QA is trying to get the cross-WG perspective via early directorate review when issues can still be easily fixed.

Of course, more reviews by experienced participants is desirable - but that's part of having a WG with energy and drafts in the design center.  Sometimes that comes by kicking (s/kicking/asking) people individually to do reviews.

Alia