Subject: Re: [payload] Some changes to rfc3984bis, SVC, and RCDO payload drafts

From: Ye-Kui Wang <[email-masked]>
Date: Sun, 20 Mar 2011 08:24:24 +0000

Folks,

The three H.264/AVC related payload formats, namely, draft-ietf-avt-rtp-rfc3984bis-12, draft-ietf-avt-rtp-svc-27, and draft-ietf-avt-rtp-h264-rcdo-08, are all at the AUTH48 stage.

The RFC-Editor has found the following problem: In draft-ietf-avt-rtp-rfc3984bis-12, the definition of the max-dpb media parameter refers to the MaxDPB that was defined the first version of the H.264/AVC spec, but not any more in the latest version (the 2010 version). The parameter in the latest H.264/AVC version corresponding to MaxDPB is MaxDpbMbs, and the unit of the new parameter (i.e., macroblocks) is different from the original one (i.e. 1024 bytes).

The problem applies also to the SVC payload format, the RCDO payload format, and H.241.

A solution has been found and agreed, involving rfc3984bis authors and some key people related to H.264/AVC (e.g., Gary Sullivan and Heiko Schwarz) and H.241 (e.g., Stephen Botzko and Patrick Luthi). Furthermore, we have found that there are also a couple of places that need fixes due to similar changes from the initial version of H.264/AVC to the latest version.

Per Roni’s suggestion, I am sending in below the changes to draft-ietf-avt-rtp-rfc3984bis-12 for review by the Payload and AVTcore WGs. It seems that exactly the same changes are needed to draft-ietf-avt-rtp-h264-rcdo-08 (co-authors of this draft may confirm), and similar but slightly different changes are needed to draft-ietf-avt-rtp-svc-27.

Since the drafts are at the AUTH48 stage, please provide comments by Monday, March 21, if any. Many thanks!

BR, YK

------------------------------------Start of suggested changes --------------------------------------

Section 8.1:
OLD:
      profile-level-id:
         A base16 [7] (hexadecimal) representation of the following
         three bytes in the sequence parameter set NAL unit is specified
         in [1]: 1) profile_idc, 2) a byte herein referred to as
         profile-iop, composed of the values of constraint_set0_flag,
         constraint_set1_flag,constraint_set2_flag,
         constraint_set3_flag, and reserved_zero_4bits in bit-
         significance order, starting from the most-significant bit, and
         3) level_idc.  Note that reserved_zero_4bits is required to be
         equal to 0 in [1], but other values for it may be specified in
         the future by ITU-T or ISO/IEC.
 NEW:
      profile-level-id:
         A base16 [7] (hexadecimal) representation of the following
         three bytes in the sequence parameter set NAL unit is specified
         in [1]: 1) profile_idc, 2) a byte herein referred to as
         profile-iop, composed of the values of constraint_set0_flag,
         constraint_set1_flag,constraint_set2_flag,
         constraint_set3_flag, constraint_set4_flag, constraint_set5_flag,
　　　　 and reserved_zero_2bits in bit-significance order, starting from the most-significant bit, and
         3) level_idc.  Note that reserved_zero_2bits is required to be
         equal to 0 in [1], but other values for it may be specified in
         the future by ITU-T or ISO/IEC.

OLD:
         For example, in the table above, profile_idc equal to 58
         (Extended) with profile-iop equal to 11xx0000 indicates the
         same sub-profile corresponding to profile_idc equal to 42
         (Baseline) with profile-iop equal to x1xx0000.  Note that other
         combinations of profile_idc and profile-iop (not listed in
         Table 5) may represent a sub-profile equivalent to the common
         subset of coding tools for more than one profile.  Note also
         that a decoder conforming to a certain profile may be able to
         decode bitstreams conforming to other profiles.  For example, a
         decoder conforming to the High 4:4:4 profile, at a certain
         level, must be able to decode bitstreams conforming to the
         Constrained Baseline, Main, High, High 10, or High 4:2:2
         profile at the same or a lower level.
 NEW:
         For example, in the table above, profile_idc equal to 58
         (Extended) with profile-iop equal to 11xx0000 indicates the
         same sub-profile corresponding to profile_idc equal to 42
         (Baseline) with profile-iop equal to x1xx0000.  Note that other
         combinations of profile_idc and profile-iop (not listed in
         Table 5) may represent a sub-profile equivalent to the common
         subset of coding tools for more than one profile.  Note also
         that a decoder conforming to a certain profile may be able to
         decode bitstreams conforming to other profiles.

OLD:
         If the profile-level-id parameter is used for capability
         exchange or session setup procedure, it indicates the subset of
         coding tools, which is equal to the default sub-profile, that
         the codec supports for both receiving and sending.
 NEW:
         If the profile-level-id parameter is used for capability
         exchange or session setup, it indicates the subset of
         coding tools, which is equal to the default sub-profile, that
         the codec supports for both receiving and sending.

OLD:
      max-cpb: The value of max-cpb is an integer indicating the maximum
         coded picture buffer size in units of 1000 bits for the VCL HRD
         parameters (see A.3.1, item i of [1]) and in units of 1200 bits
         for the NAL HRD parameters (see A.3.1, item j of [1]).
NEW:
      max-cpb: The value of max-cpb is an integer indicating the maximum
         coded picture buffer size. For Constrained Baseline, Baseline, Main 
         and Extended profiles, the unit is 1000 bits for the VCL HRD
         parameters and 1200 bits for the NAL HRD parameters. For High, High 10,
         High 4:2:2, High 4:4:4 Predictive, High 10 Intra, High 4:2:2 Intra, 
         High 4:4:4 Intra and CAVLC 4:4:4 Intra profiles, the unit is cpbBrVclFactor
         bits for the VCL HRD and cpbBrNalFactor bits for the NAL HRD parameters.

OLD:
      max-dpb: The value of max-dpb is an integer indicating the maximum
         decoded picture buffer size in units of 1024 bytes.  The max-
         dpb parameter signals that the receiver has more memory than
         the minimum amount of decoded picture buffer memory required by
         the signaled highest level conveyed in the value of the
         profile-level-id parameter or the max-recv-level parameter.
         When max-dpb is signaled, the receiver MUST be able to decode
         NAL unit streams that conform to the signaled highest level,
         with the exception that the MaxDPB value in Table A-1 of [1]
         for the signaled highest level is replaced with the value of
         max-dpb.  Consequently, a receiver that signals max-dpb MUST be
         capable of storing the following number of decoded frames,
         complementary field pairs, and non-paired fields in its decoded
         picture buffer:

            Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs *
            256 * ChromaFormatFactor ), 16)

         PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are
         defined in [1].

         The value of max-dpb MUST be greater than or equal to the value
         of MaxDPB given in Table A-1 of [1] for the highest level.
         Senders MAY use this knowledge to construct coded video streams
         with improved compression.
NEW:
      max-dpb: The value of max-dpb is an integer indicating the maximum
         decoded picture buffer size in units of 8/3 macroblocks.  The max-
         dpb parameter signals that the receiver has more memory than
         the minimum amount of decoded picture buffer memory required by
         the signaled highest level conveyed in the value of the
         profile-level-id parameter or the max-recv-level parameter.
         When max-dpb is signaled, the receiver MUST be able to decode
         NAL unit streams that conform to the signaled highest level,
         with the exception that the MaxDpbMbs value in Table A-1 of [1]
         for the signaled highest level is replaced with the value of
         max-dpb * 3 / 8.  Consequently, a receiver that signals max-dpb MUST be
         capable of storing the following number of decoded frames,
         complementary field pairs, and non-paired fields in its decoded
         picture buffer:

            Min(max-dpb * 3 / 8 / ( PicWidthInMbs * FrameHeightInMbs), 16)

         Wherein PicWidthInMbs and FrameHeightInMbs are defined in [1].

         The value of max-dpb MUST be greater than or equal to the value
         of MaxDpbMbs * 3 / 8, wherein the value of MaxDpbMbs is given in 
         Table A-1 of [1] for the highest level.
         Senders MAY use this knowledge to construct coded video streams
         with improved compression.

OLD:
      max-br: The value of max-br is an integer indicating the maximum
         video bitrate in units of 1000 bits per second for the VCL HRD
         parameters (see A.3.1, item i of [1]) and in units of 1200 bits
         per second for the NAL HRD parameters (see A.3.1, item j of
         [1]).
         …
         For example, if a receiver signals capability for Level 1.2
         with max-br equal to 1550, this indicates a maximum video
         bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum
         video bitrate of 1860 kbits/sec for NAL HRD parameters, and a
         CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000).
NEW: 
      max-br: The value of max-br is an integer indicating the maximum
         video bitrate.  For Constrained Baseline, Baseline, Main 
         and Extended profiles, the unit is 1000 bits per second for the VCL HRD
         parameters and 1200 bits per second for the NAL HRD parameters. For High, High 10,
         High 4:2:2, High 4:4:4 Predictive, High 10 Intra, High 4:2:2 Intra, 
         High 4:4:4 Intra and CAVLC 4:4:4 Intra profiles, the unit is cpbBrVclFactor
         bits per second for the VCL HRD and cpbBrNalFactor bits per second for the NAL HRD parameters.
         …
         For example, if a receiver signals capability for Main profile Level 1.2
         with max-br equal to 1550, this indicates a maximum video
         bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum
         video bitrate of 1860 kbits/sec for NAL HRD parameters, and a
         CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000).

------------------------------------End of suggested changes --------------------------------------

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sat, 26 Mar 2011 22:55:36 -0500

The proposed changes to max-dpb, max-cpb and max-br sacrifice some
backward compatibility with RFC 3984 and forward compatibility with
H.264. The latter seems intentional but the former seems unintentional.
Further changes are needed to fix or clearly convey these
incompatibilities.

What was discussed and decided regarding max-dpb, max-cpb and max-br in
the ITU/IETF F2F meetings? Based on the original suggested changes (at
the end without color coding), forward compatibility with H.264 appears
to be the focus. But after "further discussions", the newest suggested
changes (with color coding) appear focused on backward compatibility
with RFC 3984, but don't fully achieve this.

The problem with RFC 3984 backward compatibility is that changing the
units of max-dpb (from 1024 bytes to 8/3 macroblocks) only works for the
4:2:0 chroma format, which is the primary format used in most profiles
and applications. However, the high profiles support other chroma
formats, and this change breaks backward compatibility for them.

For example, consider an RFC 3984 implementation that declares
max-dpb=8100 (KB) to support 3 reference frames of 720p at 4:4:4. The
proposed change would misinterpret this declaration to mean 6 reference
frames of 720p at any chroma format including 4:4:4, hence potentially
overflowing the old RFC 3984 receiver's DPB by a factor of 2.

(Note that H.264 got away with changing MaxDPB since the original 2003
version only supported 4:2:0. When the high profiles which support other
chroma formats were added in 2005, MaxDPB was explicitly redefined in
terms of 4:2:0 so it remains constant across all chroma formats. For
further clarity, MaxDpbMbs later replaced MaxDPB. But RFC 3984 never
kept up with these changes, so implementations are stuck with the 2003
definitions even for later features such as the high profiles and other
chroma formats.)

The problem with H.264 forward compatibility is that NOT changing the
units of max-cpb and max-br for the high profiles (to include the
scaling factors 1.25/3/4) obviously creates confusion and
incompatibility between H.264 and this bis draft. This seems like an
intentional change after "further discussions" decided to focus on RFC
3984 backward compatibility at the expense of H.264 forward
compatibility.

Before deciding on specific new text, the WG should decide on whether
the focus of this draft should be backward compatibility with RFC 3984
or forward compatibility with H.264. Or if the intent is to strike some
balance between the two, then clearly convey the intentional
incompatibilities in the text.

Regards,

Mo Zanaty

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ye-Kui Wang
Sent: Monday, March 21, 2011 4:11 AM
To: [email-masked]; [email-masked]
Subject: Re: [AVTCORE] Some changes to rfc3984bis, SVC,and RCDO payload
drafts

Further discussions with the same group of persons led to a decision to
stay with the unscaled units for max-br and max-cpb, thus fewer changes
are needed to the three payload formats listed in the title and H.241.
With this, the changes needed are listed below (the originally suggested
changes are dropped from this email). This time I have highlighted the
changes, and I have also described the nature the changes below. Hope
these may help understand better what have been changed, and can lead to
a quicker decision by the group, including WG chairs, and our AD.

BR, YK

------------------------------------Start of suggested changes
--------------------------------------

Section 8.1:

OLD:

      profile-level-id:

 NEW: (note that the change here is purely editorial)

      profile-level-id:

         constraint_set3_flag, constraint_set4_flag,
constraint_set5_flag,

 and reserved_zero_2bits in bit-significance order, starting from the
most-significant bit, and

OLD:

 NEW: (note that the change here is purely editorial)

OLD:

 NEW: (note that the change here is purely editorial)

OLD: 

NEW: (note that the change here is purely editorial)

         parameters and in units of 1200 bits

         for the NAL HRD parameters.

OLD:

         picture buffer:

         defined in [1].

NEW: (When this change can be considered as editorial can be discussed,
but the nature of this change as follows. On the other hand, if not
changed, then the semantics of max-dpb is simply equivalent to
unspecified, as MaxDPB and ChromaFormatFactor are not found in the
latest H.264 spec any more. Note that compared to RFC 3984, the bits on
the wire do not change.)

         decoded picture buffer size in units of 8/3 macroblocks.  The
max-

         max-dpb * 3 / 8.  Consequently, a receiver that signals max-dpb
MUST be

         picture buffer:

            Min(max-dpb * 3 / 8 / ( PicWidthInMbs * FrameHeightInMbs),
16)

         of MaxDpbMbs * 3 / 8, wherein the value of MaxDpbMbs is given
in 

OLD:

         ...

NEW: (note that the change here is purely editorial) 

      max-br: The value of max-br is an integer indicating the maximum
         video bitrate in units of 1000 bits per second for the VCL HRD
         parameters and in units of 1200 bits
         per second for the NAL HRD parameters.
         ...

         For example, if a receiver signals capability for Main profile
Level 1.2

------------------------------------End of suggested changes
--------------------------------------

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ye-Kui Wang
Sent: Sunday, March 20, 2011 4:21 AM
To: [email-masked]; [email-masked]
Subject: [AVTCORE] Some changes to rfc3984bis, SVC, and RCDO payload
drafts

Folks,

The three H.264/AVC related payload formats, namely,
draft-ietf-avt-rtp-rfc3984bis-12, draft-ietf-avt-rtp-svc-27, and
draft-ietf-avt-rtp-h264-rcdo-08, are all at the AUTH48 stage.

The RFC-Editor has found the following problem: In
draft-ietf-avt-rtp-rfc3984bis-12, the definition of the max-dpb media
parameter refers to the MaxDPB that was defined the first version of the
H.264/AVC spec, but not any more in the latest version (the 2010
version). The parameter in the latest H.264/AVC version corresponding to
MaxDPB is MaxDpbMbs, and the unit of the new parameter (i.e.,
macroblocks) is different from the original one (i.e. 1024 bytes).

The problem applies also to the SVC payload format, the RCDO payload
format, and H.241.

A solution has been found and agreed, involving rfc3984bis authors and
some key people related to H.264/AVC (e.g., Gary Sullivan and Heiko
Schwarz) and H.241 (e.g., Stephen Botzko and Patrick Luthi).
Furthermore, we have found that there are also a couple of places that
need fixes due to similar changes from the initial version of H.264/AVC
to the latest version.

Per Roni's suggestion, I am sending in below the changes to
draft-ietf-avt-rtp-rfc3984bis-12 for review by the Payload and AVTcore
WGs. It seems that exactly the same changes are needed to
draft-ietf-avt-rtp-h264-rcdo-08 (co-authors of this draft may confirm),
and similar but slightly different changes are needed to
draft-ietf-avt-rtp-svc-27.

Since the drafts are at the AUTH48 stage, please provide comments by
Monday, March 21, if any. Many thanks!

BR, YK

------------------------------------Start of suggested changes
--------------------------------------

Section 8.1:

OLD:

      profile-level-id:

 NEW:

      profile-level-id:

         constraint_set3_flag, constraint_set4_flag,
constraint_set5_flag,

 and reserved_zero_2bits in bit-significance order, starting from the
most-significant bit, and

OLD:

 NEW:

OLD:

 NEW:

OLD:

NEW:

         coded picture buffer size. For Constrained Baseline, Baseline,
Main 

         parameters and 1200 bits for the NAL HRD parameters. For High,
High 10,

High 4:4:4 Intra and CAVLC 4:4:4 Intra profiles, the unit is
cpbBrVclFactor

OLD:

         picture buffer:

         defined in [1].

NEW:

         decoded picture buffer size in units of 8/3 macroblocks.  The
max-

         max-dpb * 3 / 8.  Consequently, a receiver that signals max-dpb
MUST be

         picture buffer:

            Min(max-dpb * 3 / 8 / ( PicWidthInMbs * FrameHeightInMbs),
16)

         of MaxDpbMbs * 3 / 8, wherein the value of MaxDpbMbs is given
in 

OLD:

         ...

NEW: 

         parameters and 1200 bits per second for the NAL HRD parameters.
For High, High 10,

High 4:4:4 Intra and CAVLC 4:4:4 Intra profiles, the unit is
cpbBrVclFactor

bits per second for the VCL HRD and cpbBrNalFactor bits per second for
the NAL HRD parameters.

         ...

         For example, if a receiver signals capability for Main profile
Level 1.2

------------------------------------End of suggested changes
--------------------------------------

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 31 Mar 2011 20:55:48 -0500

I support these proposed changes.

Best regards,

Mo Zanaty

From: [email-masked] [mailto:[email-masked]] On
Behalf Of Ye-Kui Wang
Sent: Thursday, March 31, 2011 5:42 PM
To: Roni Even; [email-masked]; [email-masked]
Cc: 'Gary Sullivan'
Subject: Re: [payload] [AVTCORE] Some changes to rfc3984bis, SVC, and
RCDO payload drafts

Please find below the suggested changes for RFC3984bis with some
informative notes added to the semantics of max-cpb, max-dpb, and
max-cr. The added notes are all highlighted in yellow. There are no
other additional changes besides the addition of these notes (compared
to the previous post). These additions should address the specific
concerns expressed so far. I hope we can close the issue and let the RFC
publication process to proceed. 

As I said earlier, the changes to the SVC and RCDO payload drafts are
very similar. However, please let the authors know if you would like
those changes to be seen also on the PAYLOAD and AVTCORE mailing lists.

Section 8.1:

OLD:

      profile-level-id:

      profile-level-id:

OLD:

OLD:

OLD: 

         parameters and in units of 1200 bits for the NAL HRD
parameters. 
         Note that this parameter does not use units of cpbBrVclFactor
and 

cpbBrNALFactor (see Table A-1 in [1]).

OLD:

         picture buffer:

         defined in [1].

            Informative note: This parameter was added primarily to

            complement a similar codepoint in the ITU-T Recommendation

            H.245, so as to facilitate signaling gateway designs.  The

            decoded picture buffer stores reconstructed samples.  There

            is no relationship between the size of the decoded picture

            buffer and the buffers used in RTP, especially
de-interleaving
 and de-jitter buffers.

NEW: (The nature of this change as follows. If the semantics remain
unchanged while the reference to H.264 spec  is the 2010 version, then
the semantics of max-dpb is simply equivalent to unspecified, as MaxDPB
and ChromaFormatFactor are not found in the 2010 H.264 spec any more.
Note that compared to RFC 3984, the bits on the wire do not change hence
there is no backward compatibility issue.)

         picture buffer:

            Informative note: This parameter was added primarily to

            complement a similar codepoint in the ITU-T Recommendation

            H.245, so as to facilitate signaling gateway designs.  The

            decoded picture buffer stores reconstructed samples.  There

            is no relationship between the size of the decoded picture

            buffer and the buffers used in RTP, especially
de-interleaving
 and de-jitter buffers.

            Informative note: In RFC 3984, which this document
obsoletes, 

            the unit of this parameter was 1024 bytes.  The unit has
been 

            changed to 8/3 macroblocks in this document.  This reason
for

            this change was due to the changes from the 2003 version of
the

            H.264 specification referenced by RFC 3984 to the 2010
version

            of the H.264 specification referenced by this document,
particularly

            the changes to Table A-1 in the H.264 specification due to
addition 

            of color formats and bit depths not supported earlier.  The 

            changed semantics of this parameter keeps backward 

            compatibility to RFC 3984 and supports all profiles defined
in 

            the 2010 version of the H.264 specification.

OLD:

      max-br: The value of max-br is an integer indicating the maximum
         video bitrate in units of 1000 bits per second for the VCL HRD
         parameters and in units of 1200 bits per second for the NAL HRD
parameters.

         Note that this parameter does not use units of cpbBrVclFactor
and 

cpbBrNALFactor (see Table A-1 in [1]).

         ...

Best Regards,

Ye-Kui Wang

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ye-Kui Wang
Sent: Monday, March 28, 2011 3:03 PM
To: Roni Even; [email-masked]; [email-masked]
Cc: 'Gary Sullivan'
Subject: Re: [AVTCORE] [payload] Some changes to rfc3984bis, SVC, and
RCDO payload drafts

Hi Roni,

Yes, we can add some notes to clarify the reason for the changes and
that the changes are backward compatible to RFC 3984. If people feel
that other notes would be helpful too, please propose, I think we are
open to all such additions as long as they would be helpful for readers
and implementers. 

BR, YK

From: Roni Even [mailto:[email-masked]] 
Sent: Monday, March 28, 2011 11:21 AM
To: Ye-Kui Wang; [email-masked]; [email-masked]
Cc: 'Gary Sullivan'; 'Roni Even'; 'Mo Zanaty (mzanaty)'
Subject: RE: [AVTCORE] [payload] Some changes to rfc3984bis, SVC, and
RCDO payload drafts

YK,

Maybe it will be helpful to add the assumption about 4:2:0 8-bit support
in RFC3984 systems to the new 3984bis draft to clarify that you need to
support the new parameter if you are using other color schemes

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Roni Even
Sent: Sunday, March 27, 2011 7:24 AM
To: 'Mo Zanaty (mzanaty)'; 'Ye-Kui Wang'; [email-masked]; [email-masked]
Cc: 'Gary Sullivan'
Subject: Re: [AVTCORE] [payload] Some changes to rfc3984bis, SVC, and
RCDO payload drafts

HI Mo,

When the solution was discussed there are two assumptions

1.        Old implementations (RFC3984) expecting the maxdpb parameter
in kbyes only supported 4:2:0 8-bit and no other color because otherwise
using only Kbytes and not macro blocks will not work for them. New
implementation(RFC6184 - the assigned RFC number to 3984bis) expects a
value in macroblocks and will interpret the value in maxdpb as
macroblocks.

So see my comments inline

Roni Even

As individual

BR, YK