Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

From: Tom Kristensen <[email-masked]>
Date: Fri, 09 Aug 2013 13:52:27 +0200

A late response after summer holiday and inbox flooding season; 
comments/answers inline below.

On 06/27/2013 07:28 PM, Wang, Ye-Kui wrote:
>
> Hi Tom,
>
> Thanks for the review and the suggestions. I have some comments on the 
> suggestions.
>
> On max-fps: The following paragraph in 
> draft-kristensen-payload-rtp-h241param-00 seems to be the main reason 
> for the introduction of max-fps. Just for a better understanding, 
> could you please clarify why the receiver side would have a preferred 
> frame rate? Does that relate to rendering/display capability? If that 
> is the case, then discarding of the additional frames should be done 
> by the rendering module instead of the decoder, as conforming decoders 
> should just output frames that are indicated, by the bitstream, to be 
> output.
>
>    If the encoder chooses to send at a higher frame rate than preferred
>
>    by the receiver side, the decoder will normally discard the
>
>    additional frames after decoding them.  The transmission of the extra
>
>    frames and the processing of frames to just discard them are wasteful
>
>    and the bandwidth and processing could be used more effectively.
>

Stephan provided convincing argumentation for this in the IETF-87 
AVTCORE session. And as we know both H.264 and H.265 allow very 
high/extreme framerates for a given level, when the resolution is small 
enough. The max-fps parameter is meant to set a limit for useful and 
preferred maximal framerate.

And yes, the decoder is one thing. Render/display capability another. 
The value for max-fps might stem from display/render capability or as 
Stephan mentioned at IETF-87 - the use of video for content sharing with 
a typically high resolution/low framerate stream.

> Not relevant for H.265/HEVC payload, but for H.264/AVC payload in RFC 
> 6184, would introduction of such a parameter cause any IOP issue 
> between a legacy sender and a new receiver? I think that should be 
> analyzed in draft-kristensen-payload-rtp-h241param-00. Certainly, it 
> would also be good to clarify the above question related to 
> rendering/display capability in 
> draft-kristensen-payload-rtp-h241param-00 too.
>

Thanks, note taken! I will update the upcoming version of 
draft-kristensen-payload-rtp-h241param with the interop/backwards 
compatibility issues valid for H.264 and the rendering/display vs. 
decoder distinction.

> On max-lts: It makes sense to me, but we would need to specify a lower 
> bound of the value, such that it would at least not contradict with 
> the min tile size implied by other parameters.
>

This parameter is no longer proposed, since the signalling need is taken 
care of in the merged proposal for parallellization tools agreep upon at 
IETF-87. The minimum tile size should be limited to the lower bound on 
the tile size in H.265, text might be needed for the dec-parallel-cap.

> On max-tr and max-tc: It seems that they are saying that there are 
> scenarios where you would need more tiles than allowed by the 
> indicated level. Could you please give an (practical) example that 
> this is needed? I am asking this because in JCT-VC there had been a 
> lot of discussions after which the level limits of MaxTileRows and 
> MaxTileCols were concluded.
>

The values of maxTileRows and maxTileColumns defined in HEVC were a 
compromise, mainly between people who wanted to limited the maximum 
number of tiles for their single-core decoder design and people who 
wanted a large number of tiles for their multi-core encoder design. 
Moreover, when making maxTileRows/Cols in HEVC level-dependent (rather 
than resolution-dependent) it was already anticipated that additional 
flexibility could be achieved during call negotiation. (See document 
JCTVC-J0235.)

There are at least three reasons for adding max-tr and max-tc as 
additional optional parameters:

    1)  Typically video conferencing uses bit rates that are 
significantly lower than the values specified in HEVC for a given 
level/resolution. For example, it is expected that 1080p30 can use bit 
rates at 1 mbps or below, while HEVC specifies a maximum bitrate of 12 
mbps for 1080p30 (level 4). To avoid designing CABAC decoding for 
unnecessarily high bit rates, the practical solution that has been used 
in the industry since H.264 days has been to signal support for a low 
level, and then send optional parameters to indicate support for a 
higher resolution. For example, a 1080p30 capable decoder could signal:
     -  max-recv-level-id = 2.1 (maxBR = 3000)
     -  max-ls = 1920x1080
     -  max-lps = 1920x0180x30
However, without being able to signal max-tc and max-tr, the encoder 
would have to comply with the values of maxTileRows and maxTileColumns 
of level 2.1 (maxTileRows = maxTileColums = 1).

    2)  Even if a decoder could signal the "true" value of 
max-recv-level-id, there are encoders who would benefit from using an 
even higher number of tiles. For example, for level 4 (1080p30), HEVC 
specifies maxTileRows = maxTileColums = 5, limiting the total number of 
tiles to 25. On the other hand, there exist processing devices having as 
many as 64 cores. Also, for sw encoder designs, it might be beneficial 
to have significantly more tiles than cores for load balancing purposes. 
A decoder that signals values of max-tr and max-tc larger than those 
indicated by the signaled level/resolution would give the encoder more 
freedom to choose the appropriate balance between number of tiles and 
level of parallelization.

    3)  Finally, for the sake of consistency, it seems reasonable 
support the same negotiation mechanisms for all parameters of HEVC 
tables A-1 and A-2, i.e. not only those corresponding to max-ls, 
max-lps, max-cbp, max-dpb, and max-br.

Cheers,
-- Tom

> BR, YK
>
> *From:* [email-masked] [mailto:[email-masked]] *On 
> Behalf Of *Tom Kristensen
> *Sent:* Thursday, June 27, 2013 6:34 AM
> *To:* Wang, Ye-Kui
> *Cc:* Geir Arne Sandbakken; [email-masked]; Tom Kristensen
> *Subject:* Re: [payload] Submission of new versions of H.265/HEVC 
> payload format
>
> We have done a first review of the new version of this draft. Seems 
> promising. However, we are missing a couple of things.
>
> First, we would like to include a max-fps parameter in line with the 
> proposed draft draft-kristensen-payload-rtp-h241param-00. A similar 
> parameter has been part of H.241 for while and will eventually be 
> valid for H.265 in H.323 land too. Note that this is a limiting 
> parameter, different from the other max-* parameters (max-ls, max-lps, 
> max-cpb, max-dpb and max-br).
>
> We propose max-fps as something along the lines of:
>
>    ------
>
>    max-fps:  The value of max-fps is an integer indicating the maximum
>
>       frame rate in units of hundredths of frames per second that can be
>
>       efficiently received.  The signaled highest level is conveyed in
>
>       the value of the profile-level-id parameter or the max-recv-level
>
>       parameter.  The max-fps parameter MAY be used to signal that the
>
>       receiver has a constraint in that it is not capable of decoding
>
>       video efficiently at the full frame rate that is implied by the
>
>       signaled highest level and, if present, one or more of the
>
>       parameters max-ls, max-lps or max-br.
>
>       Notice that the value of max-fps is not necessarily the frame rate
>
>       at which the maximum frame size can be sent, it constitutes a
>
>       constraint on maximum frame rate for all resolutions.
>
>          Informational note: The max-fps parameter is semantically
>
>          different from max-ls, max-lps, max-cpb, max-dpb,
>
>          and max-br in that max-fps is used to signal a constraint,
>
>          lowering the maximum frame rate from what is implied by the
>
>          signaled MaxLumaSR and MaxLumaPS.
>
>       The encoder MUST use a frame rate equal to or less than this
>
>       value.  In cases where the max-fps parameter is absent the
>
>       encoder is free to choose any frame rate according
>
>       to the highest signaled level and any signaled optional
>
>       parameters.
>
>    ------
>
> Second, we are missing an ability to signal support for tiles (number 
> of tiles used/to expect) and maximum tile size. The ability to signal 
> support for a number of tiles higher than what is implied for the 
> signalled level (as listed in Table A-1 in the HEVC specification. 
> These two parameters have the same semantics as the existing max-* 
> parameters (max-ls, max-lps, max-cpb, max-dpb and max-br).
>
> We propose max-tr (or max-tile-rows) and the equivalent description 
> for max-tc (or max-tile-cols) along the lines of:
>
>    ------
>
>    (Will be introduced toghether with max-ls, max-lps, max-cpb, 
> max-dpb, max-br, as parameters that "MAY be used to signal the 
> capabilities of a receiver implementation.")
>
> capabilities...:
>
>    max-tr:
>
>       The value of max-tr is an integer indication the maximum number
>
>       of tile rows. The max-tr parameter signals that the receiver is
>
>       capable of decoding video with a larger number of tile rows than
>
>       is required by the signaled highest level.
>
>       When max-tr is signaled, the receiver must be able to decode NAL
>
>       unit streams that conform to the signaled highest level, with the
>
>       exception that the MaxTileRows value in Table A-1 of [HEVC] for
>
>       the signaled highest level is replaced with the value of max-tr.
>
>       The value of max-tr MUST be greater than or equal to the value of
>
>       MaxTileRows given in Table A-1 of [HEVC] for the highest
>
>       level. Senders MAY use this knowledge to send pictures utilizing
>
>       a larger number of tile rows than is indicated in the signaled
>
>       highest level.
>
>    max-tc:
>
>       (As for max-tr with s/max-tr/max-tc/
>
>                           s/MaxTileRows/MaxTileCols/
>
>                           s/rows/colums)
>
>    ------
>
> The last tile related parameter is a new parameter named max-lts to 
> specify maximum luma tile size to make sure the decoder doesn't 
> receive a stream with too large resolution and/or bitrate in one 
> single tile. The max-lts parameter is a limiting parameter - as 
> described for max-fps above.
>
> We propose max-lts to signal the maximum size allowed in one tile 
> along the lines of:
>
>    ------
>
>    max-lts:
>
>       The value of max-lts, Maximum Luma Tile Size, is an integer
>
>       indicating the maximum tile size in units of luma samples. The
>
>       max-lts parameter signals the maximum size of one tile that the
>
>       decoder is able to decode.
>
>          Informational note: The max-fps parameter is semantically
>
>          different from max-ls, max-lps, max-cpb, max-dpb,
>
>          and max-br in that max-lts is used to signal a constraint
>
>          on the maximum tile size.
>
>       The encoder MUST use a tile size equal to or less than this
>
>       value.  In cases where the max-lts parameter is absent the
>
>       encoder is free to choose any tile size according
>
>       to the highest signaled level and any signaled optional
>
>       parameters.
>
>    ------
>
> Please consider these additions. More text needed to merge the new 
> parameters into the media subtype specification, offer/answer rules 
> and so on - of course. To be dealt with later.
>
> -- Tom
>
> On 11 June 2013 19:00, Wang, Ye-Kui <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Dear All,
>
> We have just submitted versions 02 and 03 of 
> draft-schierl-payload-rtp-h265, for which the links are as follows:
>
> Version 02:
> URL: 
> http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-02.txt
> Htmlized: http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-02
> Diff: http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-02
>
> Version 03:
> URL: 
> http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-03.txt
> Htmlized: http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-03
> Diff: http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-03
>
> Version 02 includes huge changes compared to the earlier submitted 
> version 01. In a short summary, the authors have worked hard to try to 
> make the design complete, from both the payload structure and the 
> signaling points of view. Compared to version 02, version 03 only 
> contains a correction of the email address of an author.
>
> Due to that the industry is eager to deploy H.265/HEVC and other 
> standards bodies such as 3GPP rely on the payload format for support 
> of H.265/HEVC in RTP based video services, we need to progress this 
> draft relatively quickly. Therefore, we would like to request quick 
> reviews from interested parties and also request for the WG status of 
> this draft.
>
> BR, YK (on behalf of the authors)
> _______________________________________________
> payload mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/payload
>
>
>
> -- 
> # Cisco                         | http://www.cisco.com/telepresence/
> ## [email-masked] <mailto:[email-masked]>  | 
> http://www.tandberg.com
> ###                               | http://folk.uio.no/tomkri/
>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 9 Aug 2013 17:55:46 +0000

Thanks Tom for the answers and clarifications. It seems convincing to me that we should add max-tr and max-tc on top of what have been agreed at the avtcore session in Berlin.

I will include them in the next version unless people have issues with it - so please speak up if you do have issues.

BR, YK

From: Tom Kristensen [mailto:[email-masked]]
Sent: Friday, August 09, 2013 4:52 AM
To: Wang, Ye-Kui
Cc: Tom Kristensen; [email-masked]
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

A late response after summer holiday and inbox flooding season; comments/answers inline below.

Thanks for the review and the suggestions. I have some comments on the suggestions.

On max-fps: The following paragraph in draft-kristensen-payload-rtp-h241param-00 seems to be the main reason for the introduction of max-fps. Just for a better understanding, could you please clarify why the receiver side would have a preferred frame rate? Does that relate to rendering/display capability? If that is the case, then discarding of the additional frames should be done by the rendering module instead of the decoder, as conforming decoders should just output frames that are indicated, by the bitstream, to be output.

Stephan provided convincing argumentation for this in the IETF-87 AVTCORE session. And as we know both H.264 and H.265 allow very high/extreme framerates for a given level, when the resolution is small enough. The max-fps parameter is meant to set a limit for useful and preferred maximal framerate.

And yes, the decoder is one thing. Render/display capability another. The value for max-fps might stem from display/render capability or as Stephan mentioned at IETF-87 - the use of video for content sharing with a typically high resolution/low framerate stream.

Not relevant for H.265/HEVC payload, but for H.264/AVC payload in RFC 6184, would introduction of such a parameter cause any IOP issue between a legacy sender and a new receiver? I think that should be analyzed in draft-kristensen-payload-rtp-h241param-00. Certainly, it would also be good to clarify the above question related to rendering/display capability in draft-kristensen-payload-rtp-h241param-00 too.

Thanks, note taken! I will update the upcoming version of draft-kristensen-payload-rtp-h241param with the interop/backwards compatibility issues valid for H.264 and the rendering/display vs. decoder distinction.

On max-lts: It makes sense to me, but we would need to specify a lower bound of the value, such that it would at least not contradict with the min tile size implied by other parameters.

This parameter is no longer proposed, since the signalling need is taken care of in the merged proposal for parallellization tools agreep upon at IETF-87. The minimum tile size should be limited to the lower bound on the tile size in H.265, text might be needed for the dec-parallel-cap.

On max-tr and max-tc: It seems that they are saying that there are scenarios where you would need more tiles than allowed by the indicated level. Could you please give an (practical) example that this is needed? I am asking this because in JCT-VC there had been a lot of discussions after which the level limits of MaxTileRows and MaxTileCols were concluded.

The values of maxTileRows and maxTileColumns defined in HEVC were a compromise, mainly between people who wanted to limited the maximum number of tiles for their single-core decoder design and people who wanted a large number of tiles for their multi-core encoder design. Moreover, when making maxTileRows/Cols in HEVC level-dependent (rather than resolution-dependent) it was already anticipated that additional flexibility could be achieved during call negotiation. (See document JCTVC-J0235.)

There are at least three reasons for adding max-tr and max-tc as additional optional parameters:

   1)  Typically video conferencing uses bit rates that are significantly lower than the values specified in HEVC for a given level/resolution. For example, it is expected that 1080p30 can use bit rates at 1 mbps or below, while HEVC specifies a maximum bitrate of 12 mbps for 1080p30 (level 4). To avoid designing CABAC decoding for unnecessarily high bit rates, the practical solution that has been used in the industry since H.264 days has been to signal support for a low level, and then send optional parameters to indicate support for a higher resolution. For example, a 1080p30 capable decoder could signal:
    -  max-recv-level-id = 2.1 (maxBR = 3000)
    -  max-ls = 1920x1080
    -  max-lps = 1920x0180x30
However, without being able to signal max-tc and max-tr, the encoder would have to comply with the values of maxTileRows and maxTileColumns of level 2.1 (maxTileRows = maxTileColums = 1).

   2)  Even if a decoder could signal the "true" value of max-recv-level-id, there are encoders who would benefit from using an even higher number of tiles. For example, for level 4 (1080p30), HEVC specifies maxTileRows = maxTileColums = 5, limiting the total number of tiles to 25. On the other hand, there exist processing devices having as many as 64 cores. Also, for sw encoder designs, it might be beneficial to have significantly more tiles than cores for load balancing purposes. A decoder that signals values of max-tr and max-tc larger than those indicated by the signaled level/resolution would give the encoder more freedom to choose the appropriate balance between number of tiles and level of parallelization.

   3)  Finally, for the sake of consistency, it seems reasonable support the same negotiation mechanisms for all parameters of HEVC tables A-1 and A-2, i.e. not only those corresponding to max-ls, max-lps, max-cbp, max-dpb, and max-br.

Cheers,
-- Tom

BR, YK

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Tom Kristensen
Sent: Thursday, June 27, 2013 6:34 AM
To: Wang, Ye-Kui
Cc: Geir Arne Sandbakken; [email-masked]<mailto:[email-masked]>; Tom Kristensen
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

We have done a first review of the new version of this draft. Seems promising. However, we are missing a couple of things.

First, we would like to include a max-fps parameter in line with the proposed draft draft-kristensen-payload-rtp-h241param-00. A similar parameter has been part of H.241 for while and will eventually be valid for H.265 in H.323 land too. Note that this is a limiting parameter, different from the other max-* parameters (max-ls, max-lps, max-cpb, max-dpb and max-br).

Second, we are missing an ability to signal support for tiles (number of tiles used/to expect) and maximum tile size. The ability to signal support for a number of tiles higher than what is implied for the signalled level (as listed in Table A-1 in the HEVC specification. These two parameters have the same semantics as the existing max-* parameters (max-ls, max-lps, max-cpb, max-dpb and max-br).

We propose max-tr (or max-tile-rows) and the equivalent description for max-tc (or max-tile-cols) along the lines of:
   ------
   (Will be introduced toghether with max-ls, max-lps, max-cpb, max-dpb, max-br, as parameters that "MAY be used to signal the capabilities of a receiver implementation.")
capabilities...:

   max-tr:

   max-tc:

The last tile related parameter is a new parameter named max-lts to specify maximum luma tile size to make sure the decoder doesn't receive a stream with too large resolution and/or bitrate in one single tile. The max-lts parameter is a limiting parameter - as described for max-fps above.

We propose max-lts to signal the maximum size allowed in one tile along the lines of:
   ------
   max-lts:

Please consider these additions. More text needed to merge the new parameters into the media subtype specification, offer/answer rules and so on - of course. To be dealt with later.

-- Tom

On 11 June 2013 19:00, Wang, Ye-Kui <[email-masked]<mailto:[email-masked]>> wrote:
Dear All,

We have just submitted versions 02 and 03 of draft-schierl-payload-rtp-h265, for which the links are as follows:

Version 02:
URL:             http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-02.txt
Htmlized:        http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-02
Diff:            http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-02

Version 03:
URL:             http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-03.txt
Htmlized:        http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-03
Diff:            http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-03

Version 02 includes huge changes compared to the earlier submitted version 01. In a short summary, the authors have worked hard to try to make the design complete, from both the payload structure and the signaling points of view. Compared to version 02, version 03 only contains a correction of the email address of an author.

Due to that the industry is eager to deploy H.265/HEVC and other standards bodies such as 3GPP rely on the payload format for support of H.265/HEVC in RTP based video services, we need to progress this draft relatively quickly. Therefore, we would like to request quick reviews from interested parties and also request for the WG status of this draft.

BR, YK (on behalf of the authors)
_______________________________________________
payload mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/payload

--
# Cisco                         |  http://www.cisco.com/telepresence/
## [email-masked]<mailto:[email-masked]>  |  http://www.tandberg.com
###                               |  http://folk.uio.no/tomkri/