Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

From: Rickard Sjöberg <[email-masked]>
Date: Wed, 3 Jul 2013 13:46:33 +0000

Hi Ye-Kui,

Thanks for your feedback on my comments, your suggestions look good to me.

Regarding discouraging fragmentation units (FUs) for live-encoding scenarios in section 4.7, I think using FUs does make sense when you do not have many non-VLC NAL units for aggregation. The spatial granularity of slices was 16x16 pixels in H.264 but is typically 64x64 for HEVC which means that MTU size matching is done with units that are 16x larger. This may lead to a larger number of smaller packets when slices are used compared to FUs. In addition, there is the HEVC rule of slice and tile boundaries that makes the slice granularity equal to the tile granularity for cases when slices span multiple tiles. Finally, FUs are easier to implement since you do not need to care about when to break your slice to not exceed the MTU size limit. I think both slices and FUs has their merits and the choice between them for live-encoding should be left for the implementer.

Regarding the use of PicOrderCntVal for feedback in 8.2 and 8.2 and the POC MSB sync issue, I agree that this is a corner case that we probably will not see much of in practice. I have no text change suggestion at the moment.

BR
Rickard

-----Original Message-----
From: Wang, Ye-Kui [mailto:[email-masked]] 
Sent: den 24 juni 2013 01:00
To: Rickard Sjöberg; [email-masked]
Subject: RE: Submission of new versions of H.265/HEVC payload format

Hi Rickard,

Thanks for the careful review and the comments. Please see inline below.

BR, YK

> -----Original Message-----
> From: Rickard Sjöberg [mailto:[email-masked]]
> Sent: Thursday, June 20, 2013 9:17 AM
> To: Wang, Ye-Kui; [email-masked]
> Subject: RE: Submission of new versions of H.265/HEVC payload format
> 
> Dear all,
> 
> I have taken a first look at draft-schierl-payload-rtp-h265-03 and 
> have some questions/comments.
> 
> My first question is what extensions of HEVC that 
> draft-schierl-payload-rtp-
> h265 is intended to cover? The draft mentions "future  scalable  or  
> 3D  video coding  extensions  of  this  specification", what extensions do you refer to?
> Is the intent to cover the all HEVC extensions in a single payload specification?
> 

[YK] That is in the semantics of LayerId (nuh_layer_id), where "this specification" should be replaced with "[HEVC]". It is sort of copy-and-paste error. Thanks for the good catching. No, there is no intention to cover HEVC extensions in this payload specification, though we intentionally to have the depacketization process and the use with feedback messages to work for HEVC scalability and 3DV extensions.

> Another question for my understanding is why "Receivers MUST pass 
> picture timing SEI messages and decoding unit information SEI messages 
> to the decoder"?

[YK] Generally, RTP receivers should/must pass all NAL units specified by the video coding spec to the video decoders. This is emphasized here for the two timing related SEI messages after it is said that picture output timing in RTP timestamps should be used instead (to avoid giving a wrong impression to readers that the SEI messages should be ignored). If an RTP receiver discards the SEI messages, then HRD conformance checking for the bitstream that was possible would be disabled, and also the information related to frame doubling or tripping would be lost.

> 
> In "5. Packetization Rules" in Section 4.7, there are 5 rules. The 
> fourth one does not seem to be a rule since there is no "MUST" or 
> "SHOULD" in the corresponding text.

[YK] I see your point, it is only a "MAY" rule. An alternative to put this as a packetization rule is to put it as part of the semantics of NAL unit header (1.1.4), but that sub-section belongs to an overview wherein normative language (even "MAY") should not appear. One solution to this is to insert "Payload Header Usage" after 4.1 "RTP Header Usage" and include at least this item there. I will do this in the next version unless there is a better suggestion.

> Further, the fifth one discourages using FUs, what are the reasons 
> behind that?

[YK] The bullet item only discourages using FUs for live-encoding scenarios, wherein dependent slice segments should be used instead. This was added after a discussion (among the authors) on the need of whether to specify a payload structure for mixed FU and aggregation packets and from that discussion we concluded that encoding dependent slice segments at source coding level already allows for the use cases and using of FUs for live-encoding does not make sense. Please let us know if you think differently.

> In 8.2 and 8.3, the feedback messages are using PicOrderCntVal. Isn't 
> there a possiblity that the MSB of PicOrderCntVal may differ between 
> the encoder and decoder?

[YK] Good question. In theory this is indeed possible when random accessing to a bitstream is performed. However, it would not happen in conversational applications where the feedback messages may be used. Therefore, to me this would just work just fine. But please feel free to make other suggestions.

> 
> The use of spatial-segmentation-idc may need some updates to be 
> useful, let me come back to that later on.
> 

[YK] OK. Look forward to seeing your input herein.

> 
> I have also a number of spelling suggestions for your considerations:
> 
> Section 1.1:
> Replace "community" by "communities"

[YK] I guess both are OK. But anyway I just changed it per your suggestion (for the next version).

> 
> Section 1.1.1:
> Replace "In addition, interpolation filter" by "In addition, the 
> interpolation filter"

[YK] Thanks - done.

> Replace "skipping the transform coding" by "skipping the transform"
> (transform_skip_flag of HEVC skips the transform, not the coding)
> 

 [YK] Done.

> Section 1.1.2:
> Replace:
> "2) An indication of whether there is any parameter set within the 
> current coded video sequence that updates another parameter set of the 
> same type preceding in decoding order."
> by:
> "2) An indication of whether there is no parameter set update in the 
> coded video sequence."
> since that is what no_parameter_set_update_flag in HEVC indicates.
> 

[YK] Changed the wording to "An indication of whether there is no parameter set within the current coded video sequence that updates another parameter set of the same type preceding in decoding order."

> Section 4.7:
> Replace "Type" in Figure 10, and "Type:  6 bits" by "FU Type" and "FU 
> Type:  6 bits" to avoid confusion with the Type in the payload header.
> 

[YK] Good point again. I will change the field name to be "FuType".

> Section 6:
> Replace "recovery" by "recover"

[YK] Done.

> 
> Section 7.1:
> Replace "level-id" by "level-idc"

[YK] We are consistently using "id" for profile-id, level-id, max-recv-level-id etc, similarly as in RFC 6184, instead of "idc".

> Replace "sub_layer_leve_idc[j]" by "sub_layer_level_idc[j]"

 [YK] Done.

> 
> Section 8:
> Replace "two  payload-specific  feedback  messages" by "a  
> payload-specific feedback  message", since only SPLI is "Assigned in this memo".
> Also, consider removing references to H.264 in the last paragraph of 
> Section
> 8 since the memo is about HEVC.

[YK] Good catch again. Changed.

> 
> Section 8.1:
> Seems to me that "There MUST be exactly one RPSI contained in the FCI 
> field" should be "There MUST be exactly one SPLI contained in the FCI field"

[YK] Another good catch. Changed.

> 
> 
> Best Regards,
> Rickard Sjöberg
> 
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Wang, Ye-Kui
> Sent: den 11 juni 2013 19:00
> To: [email-masked]
> Subject: [payload] Submission of new versions of H.265/HEVC payload 
> format
> 
> Dear All,
> 
> We have just submitted versions 02 and 03 of 
> draft-schierl-payload-rtp-h265, for which the links are as follows:
> 
> Version 02:
> URL:             http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-
> h265-02.txt
> Htmlized:        http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-02
> Diff:            http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-
> 02
> 
> Version 03:
> URL:             http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-
> h265-03.txt
> Htmlized:        http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-03
> Diff:            http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-
> 03
> 
> Version 02 includes huge changes compared to the earlier submitted 
> version 01. In a short summary, the authors have worked hard to try to 
> make the design complete, from both the payload structure and the 
> signaling points of view. Compared to version 02, version 03 only 
> contains a correction of the email address of an author.
> 
> Due to that the industry is eager to deploy H.265/HEVC and other 
> standards bodies such as 3GPP rely on the payload format for support 
> of H.265/HEVC in RTP based video services, we need to progress this draft relatively quickly.
> Therefore, we would like to request quick reviews from interested 
> parties and also request for the WG status of this draft.
> 
> BR, YK (on behalf of the authors)
> _______________________________________________
> payload mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/payload
>

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 3 Jul 2013 16:22:48 +0000

Hi Rickard,
Commenting only on slices vs. FUs.
The preference for slices over FUs, historically, was pushed by myself
into RFC 3984 for reasons of error resilience, and was moved over to this
draft for the same reason.  Loose a slice, and you can recover at the next
slice boundary; loose an FU, and you have to wait for the next slice
header and throw away all trailing FUs.  RTP is in virtually all cases run
over UDP, and in the typical Internet scenario UDP is lossy (in contrast
to private, managed IP-network, which may have other constraints, but are
not the IETF's main concern.)
To set your remarks into context, let's first understand what we are
talking about: the cost of slices is the overhead stemming from the
insertion of slice headers (in-picture syntax prediction) and from the
packet headers themselves.  And, of course, implementation complexity, but
we should not be in the business to encourage implementor laziness when
there would be a more complex, but better suited tool available.    The
additional packetization overhead is only incurred when, as a result of
incompetently filled packets (which, I agree, is more likely with 64x64
CUs than with 16/16 macroblocks), an additional packet needs to be
inserted.  You may remember a quick analysis of mine we discussed way back
early in JCT, where we (I think) agreed that the practical impact is
negligible in almost all non-tile scenarios (mostly because an additional
overhead of 50 bytes or so--3% or less of the coded picture size) is
incurred statistically only rarely (only in those cases where an
additional packet would need to be generated because of the video coding
related overhead for the use of slices).  The prediction overhead of the
use of regular slices has been shown to be substantial--the price one
needs to pay for independent decodability--but for entropy slices that
overhead virtually does not exist.
With respect to tiles, I think you have a point though, especially when
considering the type of massive parallel implementations for relatively
small picture sizes some folks have been considering.
OTOH, because FUs are trivial to implement--some say in contrast to
slices--and because (I hope) we all agree that using FUs in error prone
scenarios is a bad idea if you could use slices (but for reasons like the
tile connection you mentioned), the draft should IMO continue to set a
preference for the use of regular slices over FUs.  To avoid
underperforming systems due to implementer laziness.
However, this is the IETF.  We don't have to worry about the word-count of
explanatory language.  In fact, explaining such complex tradeoffs and
relationships is generally encouraged here.  So let's just do that:
express a preference of the use of regular slices (SHOULD) when you expect
losses and can use them (real-time encoding, no tiling structure that
would lead to unacceptable packetization overhead); and suggest (MAY) the
use of FUs in other scenarios.  Accompanied by a more coherently expressed
analysis.
Stephan

On 7.3.2013 06:46 , "Rickard Sjöberg" <[email-masked]> wrote:

>Regarding discouraging fragmentation units (FUs) for live-encoding
>scenarios in section 4.7, I think using FUs does make sense when you do
>not have many non-VLC NAL units for aggregation. The spatial granularity
>of slices was 16x16 pixels in H.264 but is typically 64x64 for HEVC which
>means that MTU size matching is done with units that are 16x larger. This
>may lead to a larger number of smaller packets when slices are used
>compared to FUs. In addition, there is the HEVC rule of slice and tile
>boundaries that makes the slice granularity equal to the tile granularity
>for cases when slices span multiple tiles. Finally, FUs are easier to
>implement since you do not need to care about when to break your slice to
>not exceed the MTU size limit. I think both slices and FUs has their
>merits and the choice between them for live-encoding should be left for
>the implementer.
>
>Regarding the use of PicOrderCntVal for feedback in 8.2 and 8.2 and the
>POC MSB sync issue, I agree that this is a corner case that we probably
>will not see much of in practice. I have no text change suggestion at the
>moment.
>
>BR
>Rickard
>
>
>[YK] That is in the semantics of LayerId (nuh_layer_id), where "this
>specification" should be replaced with "[HEVC]". It is sort of
>copy-and-paste error. Thanks for the good catching. No, there is no
>intention to cover HEVC extensions in this payload specification, though
>we intentionally to have the depacketization process and the use with
>feedback messages to work for HEVC scalability and 3DV extensions.
>
>[YK] Generally, RTP receivers should/must pass all NAL units specified by
>the video coding spec to the video decoders. This is emphasized here for
>the two timing related SEI messages after it is said that picture output
>timing in RTP timestamps should be used instead (to avoid giving a wrong
>impression to readers that the SEI messages should be ignored). If an RTP
>receiver discards the SEI messages, then HRD conformance checking for the
>bitstream that was possible would be disabled, and also the information
>related to frame doubling or tripping would be lost.
>
>> 
>[YK] I see your point, it is only a "MAY" rule. An alternative to put
>this as a packetization rule is to put it as part of the semantics of NAL
>unit header (1.1.4), but that sub-section belongs to an overview wherein
>normative language (even "MAY") should not appear. One solution to this
>is to insert "Payload Header Usage" after 4.1 "RTP Header Usage" and
>include at least this item there. I will do this in the next version
>unless there is a better suggestion.
>
>[YK] The bullet item only discourages using FUs for live-encoding
>scenarios, wherein dependent slice segments should be used instead. This
>was added after a discussion (among the authors) on the need of whether
>to specify a payload structure for mixed FU and aggregation packets and
>from that discussion we concluded that encoding dependent slice segments
>at source coding level already allows for the use cases and using of FUs
>for live-encoding does not make sense. Please let us know if you think
>differently.
>
>[YK] Good question. In theory this is indeed possible when random
>accessing to a bitstream is performed. However, it would not happen in
>conversational applications where the feedback messages may be used.
>Therefore, to me this would just work just fine. But please feel free to
>make other suggestions.
>
>> 
>[YK] Changed the wording to "An indication of whether there is no
>parameter set within the current coded video sequence that updates
>another parameter set of the same type preceding in decoding order."
>
>> Section 4.7:
>[YK] We are consistently using "id" for profile-id, level-id,
>max-recv-level-id etc, similarly as in RFC 6184, instead of "idc".
>
>> URL:            
>>http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-
>> h265-02.txt
>> Htmlized:       
>>http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-02
>> Diff:           
>>http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-
>> 02
>> 
>> Version 03:
>> URL:            
>>http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-
>> h265-03.txt
>> Htmlized:       
>>http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-03
>> Diff:           
>>http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-
>> 03
>>

---

From: Thomas Davies <[email-masked]>
Date: Wed, 03 Jul 2013 17:33:47 +0100

Hi Stephan

I think I agree with Rickard.

I think the text in question states a preference for _dependent_ slice 
segments, which have no error resiliency advantage over FUs. Is this 
just a typo?

<snip>

FUs SHOULD NOT be applied in live-encoding scenarios such as
       video telephony, video conferencing, live streaming and live
       broadcast, in which cases dependent slice segments SHOULD be used
       when a slice should be transported in multiple RTP packets.

</snip>

In general FUs may be used where error conditions are known to be 
benign, for greater efficiency, and also where for whatever reason the 
encoder cannot support MTU matching.

We also are formulating some more general comments on this area which we 
hope to provide more fully soon.

best regards

Thomas

On 03/07/13 17:22, Stephan Wenger wrote:
> On 7.3.2013 06:46 , "Rickard Sjöberg"<[email-masked]>  wrote:
>
>> Hi Ye-Kui,
>>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 3 Jul 2013 18:42:53 +0000

Hi All,

I drafted an email replying directly to Rickard but did not finish and there was a meeting (and then Stephan's and Thomas' emails came).

I wanted to comment that Rickard was comparing the use of FUs vs slices, not FUs vs dependent slices, while the recommendation in the draft is recommending use of dependent slices over FUs in live-encoding scenarios (so this answers Thomas question: the intention was dependent slices ). 

In any case, it does not make much sense to try MTU size matching and at the same time use either dependent slices or FUs, because dependent slices and FUs should not be used when there are considerable packet losses, while MTU size matching should be applied when there are considerable packet losses.

For live encoding, whenever splitting of a regular slice is needed, the splitting can be done by the encoder using dependent slices - and the encoder has more knowledge and is more powerful than the RTP packetizer and thus can do a better job for the splitting. 

BR, YK

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Thomas Davies
> Sent: Wednesday, July 03, 2013 9:34 AM
> To: Stephan Wenger
> Cc: [email-masked]
> Subject: Re: [payload] Submission of new versions of H.265/HEVC payload
> format
> 
> Hi Stephan
> 
> I think the text in question states a preference for _dependent_ slice segments,
> which have no error resiliency advantage over FUs. Is this just a typo?
> 
> <snip>
> 
> In general FUs may be used where error conditions are known to be benign, for
> greater efficiency, and also where for whatever reason the encoder cannot
> support MTU matching.
> 
> > into RFC 3984 for reasons of error resilience, and was moved over to
> > this draft for the same reason.  Loose a slice, and you can recover at
> > the next slice boundary; loose an FU, and you have to wait for the
> > next slice header and throw away all trailing FUs.  RTP is in
> > virtually all cases run over UDP, and in the typical Internet scenario
> > UDP is lossy (in contrast to private, managed IP-network, which may
> > have other constraints, but are not the IETF's main concern.) To set
> > your remarks into context, let's first understand what we are talking
> > about: the cost of slices is the overhead stemming from the insertion
> > of slice headers (in-picture syntax prediction) and from the packet
> > headers themselves.  And, of course, implementation complexity, but we
> > should not be in the business to encourage implementor laziness when
> > additional packetization overhead is only incurred when, as a result
> > of incompetently filled packets (which, I agree, is more likely with
> > 64x64 CUs than with 16/16 macroblocks), an additional packet needs to
> > be inserted.  You may remember a quick analysis of mine we discussed
> > way back early in JCT, where we (I think) agreed that the practical
> > impact is negligible in almost all non-tile scenarios (mostly because
> > an additional overhead of 50 bytes or so--3% or less of the coded
> > picture size) is incurred statistically only rarely (only in those
> > cases where an additional packet would need to be generated because of
> > the video coding related overhead for the use of slices).  The
> > prediction overhead of the use of regular slices has been shown to be
> > substantial--the price one needs to pay for independent
> > decodability--but for entropy slices that overhead virtually does not exist.
> > With respect to tiles, I think you have a point though, especially
> > when considering the type of massive parallel implementations for
> > relatively small picture sizes some folks have been considering.
> > slices--and because (I hope) we all agree that using FUs in error
> > prone scenarios is a bad idea if you could use slices (but for reasons
> > like the tile connection you mentioned), the draft should IMO continue
> > to set a preference for the use of regular slices over FUs.  To avoid
> > However, this is the IETF.  We don't have to worry about the
> > word-count of explanatory language.  In fact, explaining such complex
> > tradeoffs and relationships is generally encouraged here.  So let's just do that:
> > express a preference of the use of regular slices (SHOULD) when you
> > expect losses and can use them (real-time encoding, no tiling
> > structure that would lead to unacceptable packetization overhead); and
> > suggest (MAY) the use of FUs in other scenarios.  Accompanied by a
> > more coherently expressed analysis.
> > Stephan
> >
> >
> >> scenarios in section 4.7, I think using FUs does make sense when you
> >> do not have many non-VLC NAL units for aggregation. The spatial
> >> granularity of slices was 16x16 pixels in H.264 but is typically
> >> 64x64 for HEVC which means that MTU size matching is done with units
> >> that are 16x larger. This may lead to a larger number of smaller
> >> packets when slices are used compared to FUs. In addition, there is
> >> the HEVC rule of slice and tile boundaries that makes the slice
> >> granularity equal to the tile granularity for cases when slices span
> >> multiple tiles. Finally, FUs are easier to implement since you do not
> >> need to care about when to break your slice to not exceed the MTU
> >> size limit. I think both slices and FUs has their merits and the
> >> choice between them for live-encoding should be left for the implementer.
> >>
> >> Regarding the use of PicOrderCntVal for feedback in 8.2 and 8.2 and
> >> the POC MSB sync issue, I agree that this is a corner case that we
> >> probably will not see much of in practice. I have no text change
> >> suggestion at the moment.
> >>
> >> BR
> >> Rickard
> >>
> >>
> >> intention to cover HEVC extensions in this payload specification,
> >> though we intentionally to have the depacketization process and the
> >> use with feedback messages to work for HEVC scalability and 3DV extensions.
> >>
> >>> picture timing SEI messages and decoding unit information SEI
> >>> messages to the decoder"?
> >> [YK] Generally, RTP receivers should/must pass all NAL units
> >> specified by the video coding spec to the video decoders. This is
> >> emphasized here for the two timing related SEI messages after it is
> >> said that picture output timing in RTP timestamps should be used
> >> instead (to avoid giving a wrong impression to readers that the SEI
> >> messages should be ignored). If an RTP receiver discards the SEI
> >> messages, then HRD conformance checking for the bitstream that was
> >> possible would be disabled, and also the information related to frame
> doubling or tripping would be lost.
> >>
> >> this as a packetization rule is to put it as part of the semantics of
> >> NAL unit header (1.1.4), but that sub-section belongs to an overview
> >> wherein normative language (even "MAY") should not appear. One
> >> solution to this is to insert "Payload Header Usage" after 4.1 "RTP
> >> Header Usage" and include at least this item there. I will do this in
> >> the next version unless there is a better suggestion.
> >>
> >> scenarios, wherein dependent slice segments should be used instead.
> >> This was added after a discussion (among the authors) on the need of
> >> whether to specify a payload structure for mixed FU and aggregation
> >> packets and
> >> at source coding level already allows for the use cases and using of
> >> FUs for live-encoding does not make sense. Please let us know if you
> >> think differently.
> >>
> >>> In 8.2 and 8.3, the feedback messages are using PicOrderCntVal.
> >>> Isn't there a possiblity that the MSB of PicOrderCntVal may differ
> >>> between the encoder and decoder?
> >> accessing to a bitstream is performed. However, it would not happen
> >> in conversational applications where the feedback messages may be used.
> >> Therefore, to me this would just work just fine. But please feel free
> >> to make other suggestions.
> >>
> >>> current coded video sequence that updates another parameter set of
> >>> the same type preceding in decoding order."
> >>> by:
> >>> Seems to me that "There MUST be exactly one RPSI contained in the
> >>> FCI field" should be "There MUST be exactly one SPLI contained in
> >>> the FCI field"
> >>> version 01. In a short summary, the authors have worked hard to try
> >>> to make the design complete, from both the payload structure and the

---

From: Rickard Sjöberg <[email-masked]>
Date: Wed, 3 Jul 2013 19:37:35 +0000

Dear all,

I wrote slices and not dependent slices since the slice granularity and slice/tile rule applies to both slices and dependent slices. Sorry for the confusion. I will now use 'independent slices' and 'dependent slices' to distinguish between them.

I do not question the use of independent slices, I find them useful in some applications.

But the current RTP payload text says:

"FUs SHOULD NOT be applied in live-encoding scenarios such as
      video telephony, video conferencing, live streaming and live
      broadcast, in which cases dependent slice segments SHOULD be used
      when a slice should be transported in multiple RTP packets."

I believe that FUs are a viable option to dependent slices for the reasons I listed in my previous e-mail and think that we should not recommend dependent slices over FUs but leave it to the implementer.

BR
Rickard

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 3 juli 2013 20:43
To: Thomas Davies; Stephan Wenger
Cc: [email-masked]
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

Hi All,

BR, YK

> Behalf Of Thomas Davies
> Subject: Re: [payload] Submission of new versions of H.265/HEVC 
> payload format
> 
> Hi Stephan
> 
> segments, which have no error resiliency advantage over FUs. Is this just a typo?
> 
> <snip>
> 
> We also are formulating some more general comments on this area which 
> we hope to provide more fully soon.
> 
> best regards
> 
> Thomas
> 
> 
> > The preference for slices over FUs, historically, was pushed by 
> > myself into RFC 3984 for reasons of error resilience, and was moved 
> > over to this draft for the same reason.  Loose a slice, and you can 
> > recover at the next slice boundary; loose an FU, and you have to 
> > wait for the next slice header and throw away all trailing FUs.  RTP 
> > is in virtually all cases run over UDP, and in the typical Internet 
> > scenario UDP is lossy (in contrast to private, managed IP-network, 
> > which may have other constraints, but are not the IETF's main 
> > concern.) To set your remarks into context, let's first understand 
> > what we are talking
> > about: the cost of slices is the overhead stemming from the 
> > insertion of slice headers (in-picture syntax prediction) and from 
> > the packet headers themselves.  And, of course, implementation 
> > complexity, but we should not be in the business to encourage implementor laziness when
> > 64x64 CUs than with 16/16 macroblocks), an additional packet needs 
> > to be inserted.  You may remember a quick analysis of mine we 
> > discussed way back early in JCT, where we (I think) agreed that the 
> > practical impact is negligible in almost all non-tile scenarios 
> > (mostly because an additional overhead of 50 bytes or so--3% or less 
> > of the coded picture size) is incurred statistically only rarely 
> > (only in those cases where an additional packet would need to be 
> > generated because of the video coding related overhead for the use 
> > of slices).  The prediction overhead of the use of regular slices 
> > has been shown to be substantial--the price one needs to pay for 
> > independent decodability--but for entropy slices that overhead virtually does not exist.
> > prone scenarios is a bad idea if you could use slices (but for 
> > reasons like the tile connection you mentioned), the draft should 
> > IMO continue to set a preference for the use of regular slices over 
> > FUs.  To avoid underperforming systems due to implementer laziness.
> > word-count of explanatory language.  In fact, explaining such 
> > complex tradeoffs and relationships is generally encouraged here.  So let's just do that:
> > structure that would lead to unacceptable packetization overhead); 
> > and suggest (MAY) the use of FUs in other scenarios.  Accompanied by 
> > a more coherently expressed analysis.
> > Stephan
> >
> >
> >> Thanks for your feedback on my comments, your suggestions look good 
> >> to
> me.
> >>
> >> scenarios in section 4.7, I think using FUs does make sense when 
> >> you do not have many non-VLC NAL units for aggregation. The spatial 
> >> 64x64 for HEVC which means that MTU size matching is done with 
> >> units that are 16x larger. This may lead to a larger number of 
> >> smaller packets when slices are used compared to FUs. In addition, 
> >> there is the HEVC rule of slice and tile boundaries that makes the 
> >> slice granularity equal to the tile granularity for cases when 
> >> slices span multiple tiles. Finally, FUs are easier to implement 
> >> since you do not need to care about when to break your slice to not 
> >> exceed the MTU size limit. I think both slices and FUs has their 
> >> merits and the choice between them for live-encoding should be left for the implementer.
> >>
> >>> h265 is intended to cover? The draft mentions "future  scalable  
> >>> or 3D  video coding  extensions  of  this  specification", what 
> >>> extensions do you refer to?
> >> [YK] That is in the semantics of LayerId (nuh_layer_id), where 
> >> "this specification" should be replaced with "[HEVC]". It is sort 
> >> of copy-and-paste error. Thanks for the good catching. No, there is 
> >> no intention to cover HEVC extensions in this payload 
> >> specification, though we intentionally to have the depacketization 
> >> process and the use with feedback messages to work for HEVC scalability and 3DV extensions.
> >>
> >> [YK] I see your point, it is only a "MAY" rule. An alternative to 
> >> put this as a packetization rule is to put it as part of the 
> >> semantics of NAL unit header (1.1.4), but that sub-section belongs 
> >> to an overview wherein normative language (even "MAY") should not 
> >> appear. One solution to this is to insert "Payload Header Usage" 
> >> after 4.1 "RTP Header Usage" and include at least this item there. 
> >> I will do this in the next version unless there is a better suggestion.
> >>
> >> This was added after a discussion (among the authors) on the need 
> >> of whether to specify a payload structure for mixed FU and 
> >> aggregation packets and
> >> at source coding level already allows for the use cases and using 
> >> of FUs for live-encoding does not make sense. Please let us know if 
> >> you think differently.
> >>
> >> Therefore, to me this would just work just fine. But please feel 
> >> free to make other suggestions.
> >>
> >>> "2) An indication of whether there is no parameter set update in 
> >>> the coded video sequence."
> >>> Also, consider removing references to H.264 in the last paragraph 
> >>> of Section
> >>> From: [email-masked] [mailto:[email-masked]] 
> >>> On Behalf Of Wang, Ye-Kui
> >>> Subject: [payload] Submission of new versions of H.265/HEVC 
> >>> payload format
> >>>
> >>> Dear All,
> >>>
> >>> version 01. In a short summary, the authors have worked hard to 
> >>> try to make the design complete, from both the payload structure 
> >>> and the signaling points of view. Compared to version 02, version 
> >>> 03 only contains a correction of the email address of an author.
> >>>
> >>> standards bodies such as 3GPP rely on the payload format for 
> >>> support of H.265/HEVC in RTP based video services, we need to 
> >>> progress this draft relatively quickly.

---

From: Jonathan Lennox <[email-masked]>
Date: Wed, 3 Jul 2013 15:52:49 -0400

> In any case, it does not make much sense to try MTU size matching and at the same time use either dependent slices or FUs, because dependent slices and FUs should not be used when there are conside…

I don't (yet) understand H.265 well enough to know the details of dependent slices vs. FUs, but it's not correct to say that MTU size matching is only necessary in environments with high packet losses.

Many NATs don't correctly forward IP packets that are greater than an MTU.  When a UDP packet is fragmented, its UDP ports appear only in the first fragment of the fragmented IP packet, requiring a NAT to maintain state (to support in-order fragments) and to buffer fragments (to support out-of-order fragments) in order to forward the fragments properly.  Many NATs won't do this, despite RFC 4787.

Thus, MTU size matching is necessary if your network paths are possibly NATted, even when packet loss is negligible.

--
Jonathan Lennox
[email-masked]

---

From: "Thomas Davies (thdavies)" <[email-masked]>
Date: Wed, 3 Jul 2013 21:28:40 +0000

Hi Ye-Kui

>> For live encoding, whenever splitting of a regular slice is needed, the splitting can be done by the encoder using dependent slices - and the encoder has more knowledge and is more powerful than t…

There is some efficiency loss, and some complexity involved in matching a dependent  slice to a packet, and you are not gaining resilience. Can you explain further why this is a "better job" than simply fragmenting? FUs seem useful to me in attaining an MTU target in the simplest possible way, and it seems too strong to me to recommend that an encoder should be closely coupled with packetization in the way that the text implies.

Best regards

Thomas

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 3 Jul 2013 22:09:28 +0000

Thanks Jonathan - good to learn another use of MTU size matching.

>From functionality point of view, dependent slices do exactly the same job as FU, one in the codec level, and one in the RTP level, they can both split one slice into multiple fragments each of which can be encapsulated into one RTP packet.

BR, YK

> Of Jonathan Lennox
> Sent: Wednesday, July 03, 2013 12:53 PM
> To: Wang, Ye-Kui
> Cc: [email-masked]
> > In any case, it does not make much sense to try MTU size matching and at the
> same time use either dependent slices or FUs, because dependent slices and
> FUs should not be used when there are considerable packet losses, while MTU
> size matching should be applied when there are considerable packet losses.
> 
> I don't (yet) understand H.265 well enough to know the details of dependent
> slices vs. FUs, but it's not correct to say that MTU size matching is only
> necessary in environments with high packet losses.
> 
> Many NATs don't correctly forward IP packets that are greater than an MTU.
> When a UDP packet is fragmented, its UDP ports appear only in the first
> fragment of the fragmented IP packet, requiring a NAT to maintain state (to
> support in-order fragments) and to buffer fragments (to support out-of-order
> fragments) in order to forward the fragments properly.  Many NATs won't do
> this, despite RFC 4787.
> 
> Thus, MTU size matching is necessary if your network paths are possibly
> NATted, even when packet loss is negligible.
> 
> > For live encoding, whenever splitting of a regular slice is needed, the splitting
> can be done by the encoder using dependent slices - and the encoder has more
> knowledge and is more powerful than the RTP packetizer and thus can do a
> better job for the splitting.
> >
> > BR, YK
> >
> >
> >> I think the text in question states a preference for _dependent_
> >> slice segments, which have no error resiliency advantage over FUs. Is this
> just a typo?
> >>
> >> <snip>
> >>
> >> benign, for greater efficiency, and also where for whatever reason
> >> the encoder cannot support MTU matching.
> >>
> >>>> aggregation packets and from that discussion we concluded that
> >>>> encoding dependent slice segments at source coding level already
> >>>> allows for the use cases and using of FUs for live-encoding does
> >>>> not make sense. Please let us know if you think differently.
> >>>>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 3 Jul 2013 22:24:30 +0000

Hi Thomas,

FUs would involve some efficiency loss too, due to the FU header bits. Generating dependent slice for MTU size matching is not more difficult than generating regular slices for MTU matching, and it can be easily done when you know the MTU size and the IP/UDP/RTP/payload header size. I don't think this is a strong coupling of encoder and packetization, and any encoder that would do slice for MTU size matching does not anyway. Encoders has knowledge on what it plans to generate, include non-VLC NAL units et al, and it can thus try to generate the best-sized dependent slices to be aggregated together with non-VCL NAL units. 

In any case, since MTU size matching also has its use for other than error resilience purpose as Jonathan pointed out, then some of the use cases Rickard mentioned makes sense to me, and I now agree that we should just remove that recommendation. I will do this in the next version (unless the discussion goes further and to a different direction).

BR, YK

> Of Thomas Davies (thdavies)
> Sent: Wednesday, July 03, 2013 2:29 PM
> To: Wang, Ye-Kui
> Cc: [email-masked]
> >> For live encoding, whenever splitting of a regular slice is needed, the
> splitting can be done by the encoder using dependent slices - and the encoder
> has more knowledge and is more powerful than the RTP packetizer and thus
> can do a better job for the splitting.
> 
> There is some efficiency loss, and some complexity involved in matching a
> dependent  slice to a packet, and you are not gaining resilience. Can you
> explain further why this is a "better job" than simply fragmenting? FUs seem
> useful to me in attaining an MTU target in the simplest possible way, and it
> seems too strong to me to recommend that an encoder should be closely
> coupled with packetization in the way that the text implies.
> 
> Best regards
> 
> Thomas
>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 3 Jul 2013 22:26:18 +0000

Typo: "does not anyway" -> "does it anyway"

> Of Wang, Ye-Kui
> Sent: Wednesday, July 03, 2013 3:25 PM
> To: Thomas Davies (thdavies)
> Cc: [email-masked]
> FUs would involve some efficiency loss too, due to the FU header bits.
> Generating dependent slice for MTU size matching is not more difficult than
> generating regular slices for MTU matching, and it can be easily done when you
> know the MTU size and the IP/UDP/RTP/payload header size. I don't think this
> is a strong coupling of encoder and packetization, and any encoder that would
> do slice for MTU size matching does not anyway. Encoders has knowledge on
> what it plans to generate, include non-VLC NAL units et al, and it can thus try to
> generate the best-sized dependent slices to be aggregated together with non-
> VCL NAL units.
> 
> In any case, since MTU size matching also has its use for other than error
> resilience purpose as Jonathan pointed out, then some of the use cases Rickard
> mentioned makes sense to me, and I now agree that we should just remove
> that recommendation. I will do this in the next version (unless the discussion
> goes further and to a different direction).
> 
> BR, YK
> 
> > Behalf Of Thomas Davies (thdavies)
> > splitting can be done by the encoder using dependent slices - and the
> > encoder has more knowledge and is more powerful than the RTP
> > packetizer and thus can do a better job for the splitting.
> >
> > There is some efficiency loss, and some complexity involved in
> > matching a dependent  slice to a packet, and you are not gaining
> > resilience. Can you explain further why this is a "better job" than
> > simply fragmenting? FUs seem useful to me in attaining an MTU target
> > in the simplest possible way, and it seems too strong to me to
> > recommend that an encoder should be closely coupled with packetization in
> the way that the text implies.
> >
> > Best regards
> >
> > Thomas
> >

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 4 Jul 2013 04:50:40 +0000

Thomas, Paul and I have been working on improving packet loss resilience using fragments and tiles. I mentioned this to Stephan at IETF 86 in Orlando, and promised to have draft text ready before Berlin. Here is the proposed text along with an overview of the rationale.

Rationale:

1. Resilience to packet loss requires independent slices or tiles. Since independence requires restrictions on some coding tools, both potentially incur slight penalties in coding efficiency, with a smaller penalty for tiles.

2. Fragmentation in a higher layer like RTP can have several advantages over similar features in a codec layer like slices (independent or dependent).
a. Efficiency: RTP fragmentation has finer (byte) granularity while slices have much larger (CTB or tile) granularity. It also has less slice segment header overhead, especially for independent slices. This can impact the efficiency of the packetized stream in terms of both bits and packets per second.
b. Performance: Encoders must perform extra processing to limit slice sizes, which can impact encoder performance. Some implementations speculatively end a slice based on computed statistics (which aggravates a.), while others recode a slice if the size is exceeded (which aggravates b.).
c. Decoupling: Not all encoders will support limiting slice size to MTU size. Applications (for live or stored content) using such encoders can still support MTU limits using RTP fragmentation since it is decoupled from the encoder.
d. Multi-party: The encoded content may go to multiple receivers with different MTUs. RTP fragmentation can handle this with lightweight repacketization at middle boxes whereas slices incur heavyweight transcoding.

3. Combining fragmentation and tiles can improve resilience while retaining the above advantages, if fragments are enhanced to contain offsets (similar to IP fragmentation). This is described in the proposed draft text below, which would immediately follow section 4.7 Fragmentation Units.

Draft Text:

4.7.1 Fragmentation Units with Fragment Offsets

If a fragmented NAL unit contains tiles, its slice segment header contains offsets to the data of each tile. These offsets can be used for random access to any tile for parallel processing, region of interest extraction, and resilience to errors in other tiles. These offsets can also be used to provide resilience to packet loss, in conjunction with fragment offsets contained in two types of fragmentation units: FU-A2 and FU-A4.

Fragmentation units FU-A2 and FU-A4 contain a fragment offset before the FU payload. FU-A2 contains a 2-byte offset as shown in Figure X. FU-A4 contains a 4-byte offset as shown in Figure Y. The offset indicates the total number of bytes in all prior FU payloads of the fragmented NAL unit.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      FU-A2 NAL HDR (type=52)  |   FU header   | FU-A2 offset  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | FU-A2 offset  |     DONL (optional)           |               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               |
   |                                                               |
   |                         FU payload                            |
   |                                                               |
   |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                               :...OPTIONAL RTP padding        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Figure X   RTP payload format for FU-A2

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      FU-A4 NAL HDR (type=53)  |   FU header   | FU-A4 offset  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      FU-A4 offset                             | DONL(optional)|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | DONL(optional)|                                               |
   +-+-+-+-+-+-+-+-+         FU payload                            |
   |                                                               |
   |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                               :...OPTIONAL RTP padding        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Figure Y   RTP payload format for FU-A4

Since the offset of the first FU payload of a fragmented NAL unit is zero, and the Start (S) bit in the FU header is sufficient to indicate this, the first fragmentation unit of a fragmented NAL unit SHOULD use FU-A, but MAY use FU-A2 or FU-A4 with an offset of zero. Subsequent fragmentation units of a fragmented NAL unit MAY use FU-A2 or FU-A4 with a non-zero offset. The Start (S) bit in the FU header MUST NOT be set if the FU-A2 or FU-A4 contains a non-zero offset, and MUST be set if it contains a zero offset.

If a fragmentation unit is lost, tiles in the lost fragmentation unit are also lost. However, tiles which are successfully received in their entirety can be decoded if their slice segment header containing tile offsets is successfully received, despite lost fragmentation units. Fragment offsets in FU-A2 or FU-A4 are required in order to recover from loss of prior fragmentation units and continue decoding subsequent tiles.

The following heuristic SHOULD be applied to determine if the lost fragmentation units are all part of the same fragmented NAL unit as subsequently received fragmentation units with identical RTP timestamps and identical values of NAL unit header layer ID and temporal ID.

1. Let N be the number of missing sequence numbers between two received fragmentation units with known offsets.
2. Let P be the FU payload size of the first of the two received FUs.
3. Let D be the difference in the offsets, minus P.
   (This is the number of missing FU payload bytes.)
4. Let E be N times P.
   (This is the estimated number of missing FU payload bytes.)
5. Let ERROR be the absolute difference between D and E, divided by D.
6. If ERROR < 50%, it is likely all FUs are part of the same fragmented NAL unit, otherwise it is unlikely.

Cheers,
Mo

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rickard Sjöberg
Sent: Wednesday, July 03, 2013 3:38 PM
To: Wang, Ye-Kui; Thomas Davies (thdavies); Stephan Wenger
Cc: [email-masked]
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

Dear all,

BR
Rickard

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 3 juli 2013 20:43
To: Thomas Davies; Stephan Wenger
Cc: [email-masked]
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

Hi All,

BR, YK

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 4 Jul 2013 06:01:45 +0000

An interesting idea. Using dependent slices (again :-) together with tiles would do the trick too (each tile in one dependent slice, and then each dependent slice NAL unit in one single NAL unit packet).

Have you guys done an analysis comparing the pros and cons of the two alternative approaches?

BR, YK

> From: Mo Zanaty (mzanaty) [mailto:[email-masked]]
> Sent: Wednesday, July 03, 2013 9:51 PM
> To: Rickard Sjöberg; Wang, Ye-Kui; Thomas Davies (thdavies); Stephan Wenger;
> Paul Bright-Thomas (paubrigh)
> Cc: [email-masked]
> Subject: RE: [payload] Submission of new versions of H.265/HEVC payload
> format
> 
> Thomas, Paul and I have been working on improving packet loss resilience using
> fragments and tiles. I mentioned this to Stephan at IETF 86 in Orlando, and
> promised to have draft text ready before Berlin. Here is the proposed text
> along with an overview of the rationale.
> 
> Rationale:
> 
> 1. Resilience to packet loss requires independent slices or tiles. Since
> independence requires restrictions on some coding tools, both potentially incur
> slight penalties in coding efficiency, with a smaller penalty for tiles.
> 
> 2. Fragmentation in a higher layer like RTP can have several advantages over
> similar features in a codec layer like slices (independent or dependent).
> a. Efficiency: RTP fragmentation has finer (byte) granularity while slices have
> much larger (CTB or tile) granularity. It also has less slice segment header
> overhead, especially for independent slices. This can impact the efficiency of
> the packetized stream in terms of both bits and packets per second.
> b. Performance: Encoders must perform extra processing to limit slice sizes,
> which can impact encoder performance. Some implementations speculatively
> end a slice based on computed statistics (which aggravates a.), while others
> recode a slice if the size is exceeded (which aggravates b.).
> c. Decoupling: Not all encoders will support limiting slice size to MTU size.
> Applications (for live or stored content) using such encoders can still support
> MTU limits using RTP fragmentation since it is decoupled from the encoder.
> d. Multi-party: The encoded content may go to multiple receivers with different
> MTUs. RTP fragmentation can handle this with lightweight repacketization at
> middle boxes whereas slices incur heavyweight transcoding.
> 
> 3. Combining fragmentation and tiles can improve resilience while retaining the
> above advantages, if fragments are enhanced to contain offsets (similar to IP
> fragmentation). This is described in the proposed draft text below, which would
> immediately follow section 4.7 Fragmentation Units.
> 
> Draft Text:
> 
> If a fragmented NAL unit contains tiles, its slice segment header contains
> offsets to the data of each tile. These offsets can be used for random access to
> any tile for parallel processing, region of interest extraction, and resilience to
> errors in other tiles. These offsets can also be used to provide resilience to
> packet loss, in conjunction with fragment offsets contained in two types of
> fragmentation units: FU-A2 and FU-A4.
> 
> Fragmentation units FU-A2 and FU-A4 contain a fragment offset before the FU
> payload. FU-A2 contains a 2-byte offset as shown in Figure X. FU-A4 contains a
> 4-byte offset as shown in Figure Y. The offset indicates the total number of
> bytes in all prior FU payloads of the fragmented NAL unit.
> 
> 
>     0                   1                   2                   3
> Since the offset of the first FU payload of a fragmented NAL unit is zero, and
> the Start (S) bit in the FU header is sufficient to indicate this, the first
> fragmentation unit of a fragmented NAL unit SHOULD use FU-A, but MAY use
> FU-A2 or FU-A4 with an offset of zero. Subsequent fragmentation units of a
> fragmented NAL unit MAY use FU-A2 or FU-A4 with a non-zero offset. The Start
> (S) bit in the FU header MUST NOT be set if the FU-A2 or FU-A4 contains a non-
> zero offset, and MUST be set if it contains a zero offset.
> 
> If a fragmentation unit is lost, tiles in the lost fragmentation unit are also lost.
> However, tiles which are successfully received in their entirety can be decoded
> if their slice segment header containing tile offsets is successfully received,
> despite lost fragmentation units. Fragment offsets in FU-A2 or FU-A4 are
> required in order to recover from loss of prior fragmentation units and continue
> decoding subsequent tiles.
> 
> The following heuristic SHOULD be applied to determine if the lost
> fragmentation units are all part of the same fragmented NAL unit as
> subsequently received fragmentation units with identical RTP timestamps and
> identical values of NAL unit header layer ID and temporal ID.
> 
>    (This is the number of missing FU payload bytes.) 4. Let E be N times P.
>    (This is the estimated number of missing FU payload bytes.) 5. Let ERROR be
> the absolute difference between D and E, divided by D.
> Of Rickard Sjöberg
> I wrote slices and not dependent slices since the slice granularity and slice/tile
> rule applies to both slices and dependent slices. Sorry for the confusion. I will
> now use 'independent slices' and 'dependent slices' to distinguish between
> them.
> 
> I believe that FUs are a viable option to dependent slices for the reasons I listed
> in my previous e-mail and think that we should not recommend dependent
> slices over FUs but leave it to the implementer.
> 
> BR
> Rickard
> 
> 
> I drafted an email replying directly to Rickard but did not finish and there was a
> meeting (and then Stephan's and Thomas' emails came).
> 
> I wanted to comment that Rickard was comparing the use of FUs vs slices, not
> FUs vs dependent slices, while the recommendation in the draft is
> recommending use of dependent slices over FUs in live-encoding scenarios (so
> this answers Thomas question: the intention was dependent slices ).
>

---

From: "Thomas Davies (thdavies)" <[email-masked]>
Date: Thu, 4 Jul 2013 08:48:55 +0000

Yes, we did think about one tile per packet or N tiles per packet, also: I guess that is where we started from. Then you could indeed align missing tiles with missing sequence numbers. 

The issue is that you need to guarantee that you are doing this for it to be useful, and the decoder needs to know that this is guaranteed through some form of other signalling. However, the bitstream chunk associated to a tile is extremely variable, and many tile chunks would be very small. So there is significant packetization overhead for them if there is one tile per packet. 

Having N>1 reduces this overhead, but it is still there and as N increases you run the risk of breaking the MTU size which then requires that you have to re-encode to fit things in and meet the guarantee. There is definitely no time to re-encode whole tiles, especially if you are already using them for parallelization. Thus you might have to break the guarantee and fragment the packet anyhow, losing the resilience.
 
The advantage that we see with the proposal is that you can get fairly similar resilience to regular slice MTU-size matching, but with much lower overheads as you can have a single regular slice header per frame, yet have a "dumb" packetization process and no need to re-encode to hit MTU targets. In H264 MTU matching usually means re-encoding just one or two 16x16 MBs. In H265 it involves re-encoding at least one 64x64 CTB, and if you are doing tiles then you can get tiny "dangling slices" at the end of a tile, which means more small packets.

Best regards

Thomas

-----Original Message-----
From: Wang, Ye-Kui [mailto:[email-masked]] 
Sent: 04 July 2013 07:02
To: Mo Zanaty (mzanaty); Rickard Sjöberg; Thomas Davies (thdavies); Stephan Wenger; Paul Bright-Thomas (paubrigh)
Cc: [email-masked]
Subject: RE: [payload] Submission of new versions of H.265/HEVC payload format

BR, YK

> To: Rickard Sjöberg; Wang, Ye-Kui; Thomas Davies (thdavies); Stephan 
> Wenger; Paul Bright-Thomas (paubrigh)
> Cc: [email-masked]
> Subject: RE: [payload] Submission of new versions of H.265/HEVC 
> payload format
> 
> Thomas, Paul and I have been working on improving packet loss 
> resilience using fragments and tiles. I mentioned this to Stephan at 
> IETF 86 in Orlando, and promised to have draft text ready before 
> Berlin. Here is the proposed text along with an overview of the rationale.
> 
> Rationale:
> 
> 1. Resilience to packet loss requires independent slices or tiles. 
> Since independence requires restrictions on some coding tools, both 
> potentially incur slight penalties in coding efficiency, with a smaller penalty for tiles.
> 
> 2. Fragmentation in a higher layer like RTP can have several 
> advantages over similar features in a codec layer like slices (independent or dependent).
> a. Efficiency: RTP fragmentation has finer (byte) granularity while 
> slices have much larger (CTB or tile) granularity. It also has less 
> slice segment header overhead, especially for independent slices. This 
> can impact the efficiency of the packetized stream in terms of both bits and packets per second.
> b. Performance: Encoders must perform extra processing to limit slice 
> sizes, which can impact encoder performance. Some implementations 
> speculatively end a slice based on computed statistics (which 
> aggravates a.), while others recode a slice if the size is exceeded (which aggravates b.).
> Applications (for live or stored content) using such encoders can 
> still support MTU limits using RTP fragmentation since it is decoupled from the encoder.
> d. Multi-party: The encoded content may go to multiple receivers with 
> different MTUs. RTP fragmentation can handle this with lightweight 
> repacketization at middle boxes whereas slices incur heavyweight transcoding.
> 
> 3. Combining fragmentation and tiles can improve resilience while 
> retaining the above advantages, if fragments are enhanced to contain 
> offsets (similar to IP fragmentation). This is described in the 
> proposed draft text below, which would immediately follow section 4.7 Fragmentation Units.
> 
> Draft Text:
> 
> If a fragmented NAL unit contains tiles, its slice segment header 
> contains offsets to the data of each tile. These offsets can be used 
> for random access to any tile for parallel processing, region of 
> interest extraction, and resilience to errors in other tiles. These 
> offsets can also be used to provide resilience to packet loss, in 
> conjunction with fragment offsets contained in two types of fragmentation units: FU-A2 and FU-A4.
> 
> Fragmentation units FU-A2 and FU-A4 contain a fragment offset before 
> the FU payload. FU-A2 contains a 2-byte offset as shown in Figure X. 
> FU-A4 contains a 4-byte offset as shown in Figure Y. The offset 
> indicates the total number of bytes in all prior FU payloads of the fragmented NAL unit.
> 
> 
>     0                   1                   2                   3
> Since the offset of the first FU payload of a fragmented NAL unit is 
> zero, and the Start (S) bit in the FU header is sufficient to indicate 
> this, the first fragmentation unit of a fragmented NAL unit SHOULD use 
> FU-A, but MAY use
> FU-A2 or FU-A4 with an offset of zero. Subsequent fragmentation units 
> of a fragmented NAL unit MAY use FU-A2 or FU-A4 with a non-zero 
> offset. The Start
> (S) bit in the FU header MUST NOT be set if the FU-A2 or FU-A4 
> contains a non- zero offset, and MUST be set if it contains a zero offset.
> 
> However, tiles which are successfully received in their entirety can 
> be decoded if their slice segment header containing tile offsets is 
> successfully received, despite lost fragmentation units. Fragment 
> offsets in FU-A2 or FU-A4 are required in order to recover from loss 
> of prior fragmentation units and continue decoding subsequent tiles.
> 
> subsequently received fragmentation units with identical RTP 
> timestamps and identical values of NAL unit header layer ID and temporal ID.
> 
> 1. Let N be the number of missing sequence numbers between two 
> received fragmentation units with known offsets.
>    (This is the estimated number of missing FU payload bytes.) 5. Let 
> ERROR be the absolute difference between D and E, divided by D.
> 6. If ERROR < 50%, it is likely all FUs are part of the same 
> fragmented NAL unit, otherwise it is unlikely.
> 
> Cheers,
> Mo
> 
> Behalf Of Rickard Sjöberg
> I wrote slices and not dependent slices since the slice granularity 
> and slice/tile rule applies to both slices and dependent slices. Sorry 
> for the confusion. I will now use 'independent slices' and 'dependent 
> slices' to distinguish between them.
> 
> I believe that FUs are a viable option to dependent slices for the 
> reasons I listed in my previous e-mail and think that we should not 
> recommend dependent slices over FUs but leave it to the implementer.
> 
> BR
> Rickard
> 
> 
> I drafted an email replying directly to Rickard but did not finish and 
> there was a meeting (and then Stephan's and Thomas' emails came).
> 
> I wanted to comment that Rickard was comparing the use of FUs vs 
> slices, not FUs vs dependent slices, while the recommendation in the 
> draft is recommending use of dependent slices over FUs in 
> live-encoding scenarios (so this answers Thomas question: the intention was dependent slices ).
> 
> In any case, it does not make much sense to try MTU size matching and 
> at the same time use either dependent slices or FUs, because dependent 
> slices and FUs should not be used when there are considerable packet 
> losses, while MTU size matching should be applied when there are considerable packet losses.
> 
> For live encoding, whenever splitting of a regular slice is needed, 
> the splitting can be done by the encoder using dependent slices - and 
> the encoder has more knowledge and is more powerful than the RTP 
> > slice segments, which have no error resiliency advantage over FUs. 
> > Is this just a
> typo?
> >
> > <snip>
> >
> > We also are formulating some more general comments on this area 
> > which we hope to provide more fully soon.
> >
> > best regards
> >
> > Thomas
> >
> >
> > > myself into RFC 3984 for reasons of error resilience, and was 
> > > moved over to this draft for the same reason.  Loose a slice, and 
> > > you can recover at the next slice boundary; loose an FU, and you 
> > > have to wait for the next slice header and throw away all trailing 
> > > FUs.  RTP is in virtually all cases run over UDP, and in the 
> > > typical Internet scenario UDP is lossy (in contrast to private, 
> > > managed IP-network, which may have other constraints, but are not 
> > > the IETF's main
> > > complexity, but we should not be in the business to encourage 
> > > implementor
> laziness when
> > > additional packetization overhead is only incurred when, as a 
> > > result of incompetently filled packets (which, I agree, is more 
> > > likely with
> > > discussed way back early in JCT, where we (I think) agreed that 
> > > the practical impact is negligible in almost all non-tile 
> > > scenarios (mostly because an additional overhead of 50 bytes or 
> > > so--3% or less of the coded picture size) is incurred 
> > > statistically only rarely (only in those cases where an additional 
> > > packet would need to be generated because of the video coding 
> > > related overhead for the use of slices).  The prediction overhead 
> > > of the use of regular slices has been shown to be substantial--the 
> > > price one needs to pay for independent decodability--but for 
> > > entropy slices that overhead virtually
> does not exist.
> > > OTOH, because FUs are trivial to implement--some say in contrast 
> > > to slices--and because (I hope) we all agree that using FUs in 
> > > error prone scenarios is a bad idea if you could use slices (but 
> > > for reasons like the tile connection you mentioned), the draft 
> > > should IMO continue to set a preference for the use of regular 
> > > slices over FUs.  To avoid underperforming systems due to implementer laziness.
> > > complex tradeoffs and relationships is generally encouraged here.  
> > > So let's
> just do that:
> > > express a preference of the use of regular slices (SHOULD) when 
> > > you expect losses and can use them (real-time encoding, no tiling 
> > > and suggest (MAY) the use of FUs in other scenarios.  Accompanied 
> > > by a more coherently expressed analysis.
> > > Stephan
> > >
> > >
> > > On 7.3.2013 06:46 , "Rickard 
> > > Sjöberg"<[email-masked]>
> wrote:
> > >
> > >> Hi Ye-Kui,
> > >>
> > >> Thanks for your feedback on my comments, your suggestions look 
> > >> good to
> > me.
> > >>
> > >> Regarding discouraging fragmentation units (FUs) for 
> > >> live-encoding scenarios in section 4.7, I think using FUs does 
> > >> make sense when you do not have many non-VLC NAL units for 
> > >> aggregation. The spatial granularity of slices was 16x16 pixels 
> > >> in H.264 but is typically
> > >> smaller packets when slices are used compared to FUs. In 
> > >> addition, there is the HEVC rule of slice and tile boundaries 
> > >> that makes the slice granularity equal to the tile granularity 
> > >> for cases when slices span multiple tiles. Finally, FUs are 
> > >> easier to implement since you do not need to care about when to 
> > >> break your slice to not exceed the MTU size limit. I think both 
> > >> slices and FUs has their merits and the choice between them for 
> > >> live-encoding should be left for
> the implementer.
> > >>
> > >> Regarding the use of PicOrderCntVal for feedback in 8.2 and 8.2 
> > >> and the POC MSB sync issue, I agree that this is a corner case 
> > >> that we probably will not see much of in practice. I have no text 
> > >> change suggestion at the moment.
> > >>
> > >> BR
> > >> Rickard
> > >>
> > >>
> > >>> I have taken a first look at draft-schierl-payload-rtp-h265-03 
> > >>> and have some questions/comments.
> > >>>
> > >> of copy-and-paste error. Thanks for the good catching. No, there 
> > >> is no intention to cover HEVC extensions in this payload 
> > >> specification, though we intentionally to have the 
> > >> depacketization process and the use with feedback messages to 
> > >> work for HEVC scalability
> and 3DV extensions.
> > >>
> > >>> Another question for my understanding is why "Receivers MUST 
> > >>> pass picture timing SEI messages and decoding unit information 
> > >>> SEI messages to the decoder"?
> > >> emphasized here for the two timing related SEI messages after it 
> > >> is said that picture output timing in RTP timestamps should be 
> > >> used instead (to avoid giving a wrong impression to readers that 
> > >> the SEI messages should be ignored). If an RTP receiver discards 
> > >> the SEI messages, then HRD conformance checking for the bitstream 
> > >> that was possible would be disabled, and also the information 
> > >> related to frame
> > >>> In "5. Packetization Rules" in Section 4.7, there are 5 rules. 
> > >>> The fourth one does not seem to be a rule since there is no 
> > >>> "MUST" or "SHOULD" in the corresponding text.
> > >> semantics of NAL unit header (1.1.4), but that sub-section 
> > >> belongs to an overview wherein normative language (even "MAY") 
> > >> should not appear. One solution to this is to insert "Payload Header Usage"
> > >>> Further, the fifth one discourages using FUs, what are the 
> > >>> reasons behind that?
> > >> of FUs for live-encoding does not make sense. Please let us know 
> > >> if you think differently.
> > >>
> > >>> Isn't there a possiblity that the MSB of PicOrderCntVal may 
> > >>> differ between the encoder and decoder?
> > >> accessing to a bitstream is performed. However, it would not 
> > >> happen in conversational applications where the feedback messages may be used.
> > >>> "2) An indication of whether there is any parameter set within 
> > >>> the current coded video sequence that updates another parameter 
> > >>> set of the same type preceding in decoding order."
> > >>> by:
> > >> parameter set within the current coded video sequence that 
> > >> updates another parameter set of the same type preceding in decoding order."
> > >>
> > >>> Section 4.7:
> > >>> Also, consider removing references to H.264 in the last 
> > >>> paragraph of Section
> > >>> Seems to me that "There MUST be exactly one RPSI contained in 
> > >>> the FCI field" should be "There MUST be exactly one SPLI 
> > >>> contained in the FCI field"
> > >>> Version 02 includes huge changes compared to the earlier 
> > >>> submitted version 01. In a short summary, the authors have 
> > >>> worked hard to try to make the design complete, from both the 
> > >>> payload structure and the signaling points of view. Compared to 
> > >>> version 02, version
> > >>> Therefore, we would like to request quick reviews from 
> > >>> interested parties and also request for the WG status of this draft.
> > >>>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 4 Jul 2013 15:34:08 +0000

Hi Thomas,

My point was really about using dependent slices, not about 1 or N tiles per packets (sorry that I did not make that clearer). Whatever strategy you would like to do in terms of the number of tiles in a packet, using dependent slices can do the same, as each dependent slice can include 1 or N tiles.

I see the following advantages with the approach using dependent slices instead of defining new FUs:

- First of all, no need to define new payload structures.

- The RTP packetizer does not need to parse slice headers to identify boundaries of tiles within a coded slice, as tiles of one SLICE are already in separate NAL units each containing a DEPENDENT SLICE.

- No additional overhead in packetization (the 1-byte FU header and the 2- or 4-byte FU offset). The overhead of the short slice header for dependent slice NAL units is compensated by less number of tile entry offsets signalled, as the entry offset of the first tile in each NAL unit is not signalled in the slice header. Using dependent slices would need more two-byte NAL unit headers, while using new FUs would need more two-byte packet payload headers, and the overhead bits here are the same for both methods.

Am I missing anything here?

BR, YK

> From: Thomas Davies (thdavies) [mailto:[email-masked]]
> Sent: Thursday, July 04, 2013 1:49 AM
> To: Wang, Ye-Kui; Mo Zanaty (mzanaty); Rickard Sjöberg; Stephan Wenger;
> Yes, we did think about one tile per packet or N tiles per packet, also: I guess
> that is where we started from. Then you could indeed align missing tiles with
> missing sequence numbers.
> 
> The issue is that you need to guarantee that you are doing this for it to be
> useful, and the decoder needs to know that this is guaranteed through some
> form of other signalling. However, the bitstream chunk associated to a tile is
> extremely variable, and many tile chunks would be very small. So there is
> significant packetization overhead for them if there is one tile per packet.
> 
> Having N>1 reduces this overhead, but it is still there and as N increases you
> run the risk of breaking the MTU size which then requires that you have to re-
> encode to fit things in and meet the guarantee. There is definitely no time to
> re-encode whole tiles, especially if you are already using them for
> parallelization. Thus you might have to break the guarantee and fragment the
> packet anyhow, losing the resilience.
> 
> The advantage that we see with the proposal is that you can get fairly similar
> resilience to regular slice MTU-size matching, but with much lower overheads
> as you can have a single regular slice header per frame, yet have a "dumb"
> packetization process and no need to re-encode to hit MTU targets. In H264
> MTU matching usually means re-encoding just one or two 16x16 MBs. In H265
> it involves re-encoding at least one 64x64 CTB, and if you are doing tiles then
> you can get tiny "dangling slices" at the end of a tile, which means more small
> packets.
> 
> Best regards
> 
> Thomas
> 
> An interesting idea. Using dependent slices (again :-) together with tiles would
> do the trick too (each tile in one dependent slice, and then each dependent
> slice NAL unit in one single NAL unit packet).
>

---

From: "Thomas Davies (thdavies)" <[email-masked]>
Date: Thu, 4 Jul 2013 16:22:55 +0000

Hi Ye-Kui

Thank you for the clarification - I understand your point a bit better now.

With the proposed scheme there is no need for the RTP packetizer to parse slice headers or identify tile boundaries. All it needs to do is to keep track of the offset of the start of the payload to the beginning of the NALU and put this into the field. It needs to know this offset anyway. The packetizer is completely dumb.

The point is that a decoder wishing to do error resilience can parse the slice header and identify the tile boundaries in the case that a packet goes missing, for tiles that occur in the same frame but after the packet that is lost. The decoder can compare the tile offsets with the FU offsets to work out what data is missing.
 
Re dependent slices, there are indirect overheads as well, which come from having to quantize the packet size to a whole number of tiles, plus you have to manage the encoder to match the MTU size. The granularity of tiles you need for your parallel processing may not match the granularity you need to fit in the MTU very well.

Best regards

Thomas

-----Original Message-----
From: Wang, Ye-Kui [mailto:[email-masked]] 
Sent: 04 July 2013 16:34
To: Thomas Davies (thdavies); Mo Zanaty (mzanaty); Rickard Sjöberg; Stephan Wenger; Paul Bright-Thomas (paubrigh)
Cc: [email-masked]
Subject: RE: [payload] Submission of new versions of H.265/HEVC payload format

Hi Thomas,

BR, YK

> To: Wang, Ye-Kui; Mo Zanaty (mzanaty); Rickard Sjöberg; Stephan 
> Yes, we did think about one tile per packet or N tiles per packet, 
> also: I guess that is where we started from. Then you could indeed 
> align missing tiles with missing sequence numbers.
> 
> The issue is that you need to guarantee that you are doing this for it 
> to be useful, and the decoder needs to know that this is guaranteed 
> through some form of other signalling. However, the bitstream chunk 
> associated to a tile is extremely variable, and many tile chunks would 
> be very small. So there is significant packetization overhead for them if there is one tile per packet.
> 
> Having N>1 reduces this overhead, but it is still there and as N 
> increases you run the risk of breaking the MTU size which then 
> requires that you have to re- encode to fit things in and meet the 
> guarantee. There is definitely no time to re-encode whole tiles, 
> especially if you are already using them for parallelization. Thus you 
> might have to break the guarantee and fragment the packet anyhow, losing the resilience.
> 
> The advantage that we see with the proposal is that you can get fairly 
> similar resilience to regular slice MTU-size matching, but with much 
> lower overheads as you can have a single regular slice header per frame, yet have a "dumb"
> packetization process and no need to re-encode to hit MTU targets. In 
> H264 MTU matching usually means re-encoding just one or two 16x16 MBs. 
> In H265 it involves re-encoding at least one 64x64 CTB, and if you are 
> doing tiles then you can get tiny "dangling slices" at the end of a 
> tile, which means more small packets.
> 
> Best regards
> 
> Thomas
> 
> To: Mo Zanaty (mzanaty); Rickard Sjöberg; Thomas Davies (thdavies); 
> Stephan Wenger; Paul Bright-Thomas (paubrigh)
> Cc: [email-masked]
> An interesting idea. Using dependent slices (again :-) together with 
> tiles would do the trick too (each tile in one dependent slice, and 
> then each dependent slice NAL unit in one single NAL unit packet).
> 
> > potentially incur slight penalties in coding efficiency, with a 
> > smaller penalty
> for tiles.
> >
> > advantages over similar features in a codec layer like slices 
> > (independent or
> dependent).
> > slice segment header overhead, especially for independent slices. 
> > This can impact the efficiency of the packetized stream in terms of 
> > both bits and
> packets per second.
> > b. Performance: Encoders must perform extra processing to limit 
> > slice sizes, which can impact encoder performance. Some 
> > implementations speculatively end a slice based on computed 
> > statistics (which aggravates a.), while others recode a slice if the 
> > size is exceeded (which
> aggravates b.).
> > still support MTU limits using RTP fragmentation since it is 
> > decoupled from
> the encoder.
> > d. Multi-party: The encoded content may go to multiple receivers 
> > with different MTUs. RTP fragmentation can handle this with 
> > lightweight repacketization at middle boxes whereas slices incur heavyweight transcoding.
> >
> > proposed draft text below, which would immediately follow section 
> > 4.7
> Fragmentation Units.
> >
> > Draft Text:
> >
> > conjunction with fragment offsets contained in two types of 
> > fragmentation
> units: FU-A2 and FU-A4.
> >
> > indicates the total number of bytes in all prior FU payloads of the 
> > fragmented
> NAL unit.
> >
> >
> >     0                   1                   2                   3
> > zero, and the Start (S) bit in the FU header is sufficient to 
> > indicate this, the first fragmentation unit of a fragmented NAL unit 
> > SHOULD use FU-A, but MAY use
> > FU-A2 or FU-A4 with an offset of zero. Subsequent fragmentation 
> > units of a fragmented NAL unit MAY use FU-A2 or FU-A4 with a 
> > non-zero offset. The Start
> >    (This is the estimated number of missing FU payload bytes.) 5. 
> > Let ERROR be the absolute difference between D and E, divided by D.
> > and slice/tile rule applies to both slices and dependent slices. 
> > Sorry for the confusion. I will now use 'independent slices' and 
> > 'dependent slices' to distinguish between them.
> >
> > I drafted an email replying directly to Rickard but did not finish 
> > and there was a meeting (and then Stephan's and Thomas' emails came).
> >
> > live-encoding scenarios (so this answers Thomas question: the 
> > intention was
> dependent slices ).
> >
> > In any case, it does not make much sense to try MTU size matching 
> > and at the same time use either dependent slices or FUs, because 
> > dependent slices and FUs should not be used when there are 
> > considerable packet losses, while MTU size matching should be 
> > applied when there are
> considerable packet losses.
> >
> > the splitting can be done by the encoder using dependent slices - 
> > and the encoder has more knowledge and is more powerful than the RTP 
> > > On Behalf Of Thomas Davies
> > > > moved over to this draft for the same reason.  Loose a slice, 
> > > > and you can recover at the next slice boundary; loose an FU, and 
> > > > you have to wait for the next slice header and throw away all 
> > > > trailing FUs.  RTP is in virtually all cases run over UDP, and 
> > > > in the typical Internet scenario UDP is lossy (in contrast to 
> > > > private, managed IP-network, which may have other constraints, 
> > > > but are not the IETF's main
> > > > concern.) To set your remarks into context, let's first 
> > > > understand what we are talking
> > > > insertion of slice headers (in-picture syntax prediction) and 
> > > > from the packet headers themselves.  And, of course, 
> > > > implementation complexity, but we should not be in the business 
> > > > to encourage implementor
> > laziness when
> > > > 64x64 CUs than with 16/16 macroblocks), an additional packet 
> > > > needs to be inserted.  You may remember a quick analysis of mine 
> > > > we discussed way back early in JCT, where we (I think) agreed 
> > > > that the practical impact is negligible in almost all non-tile 
> > > > statistically only rarely (only in those cases where an 
> > > > additional packet would need to be generated because of the 
> > > > video coding related overhead for the use of slices).  The 
> > > > prediction overhead of the use of regular slices has been shown 
> > > > to be substantial--the price one needs to pay for independent 
> > > > With respect to tiles, I think you have a point though, 
> > > > especially when considering the type of massive parallel 
> > > > implementations for relatively small picture sizes some folks have been considering.
> > > > slices over FUs.  To avoid underperforming systems due to 
> > > > implementer
> laziness.
> > > > you expect losses and can use them (real-time encoding, no 
> > > > tiling structure that would lead to unacceptable packetization 
> > > > overhead); and suggest (MAY) the use of FUs in other scenarios.  
> > > > Accompanied by a more coherently expressed analysis.
> > > > Stephan
> > > >
> > > >
> > > >> that we probably will not see much of in practice. I have no 
> > > >> text change suggestion at the moment.
> > > >>
> > > >> BR
> > > >> Rickard
> > > >>
> > > >>
> > > >>> h265 is intended to cover? The draft mentions "future  
> > > >>> scalable or 3D  video coding  extensions  of  this  
> > > >>> specification", what extensions do you refer to?
> > > >> "this specification" should be replaced with "[HEVC]". It is 
> > > >> sort of copy-and-paste error. Thanks for the good catching. No, 
> > > >> there is no intention to cover HEVC extensions in this payload 
> > > >> specified by the video coding spec to the video decoders. This 
> > > >> is emphasized here for the two timing related SEI messages 
> > > >> after it is said that picture output timing in RTP timestamps 
> > > >> should be used instead (to avoid giving a wrong impression to 
> > > >> readers that the SEI messages should be ignored). If an RTP 
> > > >> receiver discards the SEI messages, then HRD conformance 
> > > >> checking for the bitstream that was possible would be disabled, 
> > > >> and also the information related to frame
> > > >> [YK] I see your point, it is only a "MAY" rule. An alternative 
> > > >> to put this as a packetization rule is to put it as part of the 
> > > >> should not appear. One solution to this is to insert "Payload 
> > > >> Header
> Usage"
> > > >> [YK] The bullet item only discourages using FUs for 
> > > >> live-encoding scenarios, wherein dependent slice segments should be used instead.
> > > >> This was added after a discussion (among the authors) on the 
> > > >> need of whether to specify a payload structure for mixed FU and 
> > > >> aggregation packets and
> > > >> at source coding level already allows for the use cases and 
> > > >> using of FUs for live-encoding does not make sense. Please let 
> > > >> us know if you think differently.
> > > >>
> > > >> [YK] Good question. In theory this is indeed possible when 
> > > >> random accessing to a bitstream is performed. However, it would 
> > > >> not happen in conversational applications where the feedback 
> > > >> messages may
> be used.
> > > >> Therefore, to me this would just work just fine. But please 
> > > >> feel free to make other suggestions.
> > > >>
> > > >>> The use of spatial-segmentation-idc may need some updates to 
> > > >>> be useful, let me come back to that later on.
> > > >>>
> > > >>> Replace "In addition, interpolation filter" by "In addition, 
> > > >>> the interpolation filter"
> > > >> [YK] Thanks - done.
> > > >>
> > > >>> the current coded video sequence that updates another 
> > > >>> parameter set of the same type preceding in decoding order."
> > > >>> by:
> > > >>> "2) An indication of whether there is no parameter set update 
> > > >>> in the coded video sequence."
> > > >> [YK] Changed the wording to "An indication of whether there is 
> > > >> no parameter set within the current coded video sequence that 
> > > >> updates another parameter set of the same type preceding in 
> > > >> decoding
> order."
> > > >>
> > > >>> Section 4.7:
> > > >>> From: [email-masked] 
> > > >>> [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
> > > >>> payload structure and the signaling points of view. Compared 
> > > >>> to version 02, version
> > > >>> Due to that the industry is eager to deploy H.265/HEVC and 
> > > >>> other standards bodies such as 3GPP rely on the payload format 
> > > >>> for support of H.265/HEVC in RTP based video services, we need 
> > > >>> to progress this draft relatively quickly.

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 4 Jul 2013 17:26:10 +0000

Hi Mo,
A few comments about some of your arguments below.
Stephan

On 7.3.2013 21:50 , "Mo Zanaty (mzanaty)" <[email-masked]> wrote:

>[Š]
>
>2. Fragmentation in a higher layer like RTP can have several advantages
>over similar features in a codec layer like slices (independent or
>dependent).
>slices have much larger (CTB or tile) granularity. It also has less slice
>segment header overhead, especially for independent slices. This can
>impact the efficiency of the packetized stream in terms of both bits and
>packets per second.

About this aspect I have had a lot of communication about with Rickard and
others.  The benefit one gets in terms of bits and packets per second is
negligible when comparing dependent slices and fragmentation (i.e. the
additional overhead stems from the finer granularity and the resulting
lower number of packets generated in certain corner cases).  I could walk
you through the math, but have to dig through old messages or recreate it.
 Obviously, when using independent slices, one gets the additional benefit
of error resilience, and pays for it through prediction penalty overhead.

>speculatively end a slice based on computed statistics (which aggravates
>a.), while others recode a slice if the size is exceeded (which
>aggravates b.).

My main comment here is that it is entirely possible to design an encoder
which implements your option a) in a fairly reliable way; it's bread and
butter technology and has been for decades.  Let me also note that, in
option b) the re-encode only applies to the entropy coding, which is a
very small part of an H.265 encoder in terms of complexity in almost any
architecture, and certainly for the bitrates used in most RTP-based
systems.  Both can have have been used in practical systems with
competitive R-D performance.  To put things into perspective, we are
talking about what?  0.1% encoder cycle increase for option a?  Less?

>c. Decoupling: Not all encoders will support limiting slice size to MTU
>size. Applications (for live or stored content) using such encoders can

Yes, this is a fine argument.  At the first glance.  However: Those
decoupled, dumb encoders, which cannot even cope with trivially
implementable features involving static conditions (like MTU size
matching) are not likely to cope well with less static, and equally
disturbing features of IP network such as packet loss.  As certain
products in the video conferencing market can tell us :-)  And yes, those
systems cope with packet loss through FEC and whatnot, but that, I would
suggest, is a bolt-on to fix a broken system rather than a conscious,
organic, design choice that was from the outset based on the environment
the system is working in.

>repacketization at middle boxes whereas slices incur heavyweight

No one argues against removing fragmentation as such, and this is one use
case where it makes sense.  Though, AFAIK, even that use case has lost
most of its relevance nowadays, where the 500 octet MTU size of 2G and
2.5G networks is a thing of the past, at least for video).
[...]
>I wrote slices and not dependent slices since the slice granularity and
>slice/tile rule applies to both slices and dependent slices. Sorry for
>the confusion. I will now use 'independent slices' and 'dependent slices'
>to distinguish between them.
>
>I wanted to comment that Rickard was comparing the use of FUs vs slices,
>not FUs vs dependent slices, while the recommendation in the draft is
>recommending use of dependent slices over FUs in live-encoding scenarios
>(so this answers Thomas question: the intention was dependent slices ).
>
>In any case, it does not make much sense to try MTU size matching and at
>the same time use either dependent slices or FUs, because dependent
>encoder has more knowledge and is more powerful than the RTP packetizer
>and thus can do a better job for the splitting.
>
>BR, YK
>
>
>>implementor laziness when

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 4 Jul 2013 17:59:12 +0000

Hi Thomas,

Thanks for the clarification - I think now understand your scheme better.

So you did not intent to encapsulate only entire tiles into packets, though that would obviously be more error resilient as you won't encounter the problem where a part of a tile gets lost and the received rest becomes useless. Rather, one tile may be carried in two (or even more) packets. 

I think I was confused by the suggested language such as "If a fragmentation unit is lost, tiles in the lost fragmentation unit are also lost. However, tiles which are successfully received in their entirety can be decoded if their slice segment header containing tile offsets is successfully received, despite lost fragmentation units.", from where I got an impression that an FU includes entire tiles.

As seen from other discussions, it is very questionable how big a penalty would be when an encoder tries to splitting encoded data into separate NAL units (slices or dependent slices) for MTU size matching, even for large CTB sizes - that's the reason why the JCT-VC later decided to remove the fine-granularity slice feature.

Let's tentatively assume that overall penalty and the overhead of the new-FU approach are roughly equivalent, and take into account latest comments from Mo and Stephan, do we see an advantage of the new-FU approach compared to using dependent slices?

I am not trying to conclude yet - just try to (very briefly) summarize the situation so far at least for myself.  We should certainly have more discussions - preferable involving more people who care about this payload format.

BR, YK

> Sent: Thursday, July 04, 2013 9:23 AM
> With the proposed scheme there is no need for the RTP packetizer to parse
> slice headers or identify tile boundaries. All it needs to do is to keep track of the
> offset of the start of the payload to the beginning of the NALU and put this into
> the field. It needs to know this offset anyway. The packetizer is completely
> dumb.
> 
> The point is that a decoder wishing to do error resilience can parse the slice
> header and identify the tile boundaries in the case that a packet goes missing,
> for tiles that occur in the same frame but after the packet that is lost. The
> decoder can compare the tile offsets with the FU offsets to work out what data
> is missing.
> 
> Re dependent slices, there are indirect overheads as well, which come from
> having to quantize the packet size to a whole number of tiles, plus you have to
> manage the encoder to match the MTU size. The granularity of tiles you need
> for your parallel processing may not match the granularity you need to fit in
> the MTU very well.
> 
> Best regards
> 
> Thomas
> 
> 
> My point was really about using dependent slices, not about 1 or N tiles per
> packets (sorry that I did not make that clearer). Whatever strategy you would
> like to do in terms of the number of tiles in a packet, using dependent slices can
> do the same, as each dependent slice can include 1 or N tiles.
> 
> - The RTP packetizer does not need to parse slice headers to identify boundaries
> of tiles within a coded slice, as tiles of one SLICE are already in separate NAL
> units each containing a DEPENDENT SLICE.
> 
> - No additional overhead in packetization (the 1-byte FU header and the 2- or
> 4-byte FU offset). The overhead of the short slice header for dependent slice
> NAL units is compensated by less number of tile entry offsets signalled, as the
> entry offset of the first tile in each NAL unit is not signalled in the slice header.
> Using dependent slices would need more two-byte NAL unit headers, while
> using new FUs would need more two-byte packet payload headers, and the
> overhead bits here are the same for both methods.
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Fri, 5 Jul 2013 04:18:37 +0000

Hi Stephan,

I agree with most of your points, in the context of "design an encoder" as you mention. However, this draft/RFC is for designers of RTP apps not encoders. H.265 itself is for designers of encoders. We can (and should) try to influence encoder designers to optimize live RTP apps. But we should also give RTP apps the best tools to optimize themselves given any encoder they use.

I think you hint at a more general point, that we should focus on joint source and channel coding. So encoders and RTP apps coordinate an optimal division of labor rather than do it all or expect the other to do it all. I also agree with this point as an ideal situation, but think we should also provide tools in tactical less-than-ideal situations.

More inline.

Cheers,
Mo

On Jul 4, 2013, at 1:26 PM, "Stephan Wenger" <[email-masked]> wrote:

Hi Mo,

Mo: I know the math well. I agree the bps delta may be negligible, but the pps delta can be significant, and wireless networks are often more stressed by pps than bps. The worst case is going from 2 to 3 packets per frame, which is 50% pps overhead.

Mo: Agreed. But the draft is about designing RTP apps not encoders.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 05 Jul 2013 03:08:08 -0400

On 7/4/2013 1:26 PM, Stephan Wenger wrote:
> Mo wrote:

Agreed, mostly - I've built systems using H.264 fragmentation without 
limiting NAL size without problems.  These tended to stay at bandwidths 
<1Mbps (and in fact almost entirely at bandwidths <384 Kbps, and thus on 
average a frame was less than one 1500-byte packet, with a few things 
like IDRs being the exceptions).  And perhaps some 
non-interactive-focused (hardware) encoders might not provide the 
options you'd want here.  But I agree in general.

> 2.5G networks is a thing of the past, at least for video).

I would note that per papers submitted at the RTP congestion control 
workshop last year, larger packets on cell networks have considerably 
higher loss rates, and so you may want to limit packet size to those 
clients even though they support 1500-byte packets, and at higher loss 
rates you also may lose less total "goodput" by running a larger number 
of smaller packets, even with overhead.

-- 
Randell Jesup
[email-masked]

---

From: Thomas Schierl <[email-masked]>
Date: Fri, 5 Jul 2013 11:34:36 +0200

Dear Ye-Kui, all,

Further I would tentatively assume that you can use the slice segment address in the dependent slice fragment header for the same degree of error resilience as the new-FU header, if you put a tile into a dependent slice. And this part would be understood by the decoder itself, which should be able to do the error resilience on codec level. Without fancy interfaces into the decoder, etc.

A point in the discussion which seems to be missing is that the rewriting of a single slice into multiple dependent slice fragments (including also one independent slice fragment) can be done at very low processing costs, as shown during the standardization of HEVC. 

Thomas 
--
Thomas Schierl
Fraunhofer HHI

---

From: Thomas Davies <[email-masked]>
Date: Fri, 05 Jul 2013 15:20:30 +0100

Dear Thomas

On your first point, the decoder always has to be told there is missing 
data. I don't see much different or fancy in the interface between the 
two cases. Usually error concealment must be addressed both within the 
codec (because of reference dependencies) and also in the application 
(because of display and call-management/error response functionality). 
That's surely a very minor issue.

On the second point, the context we are considering in our proposal is 
where FUs are being used, in the circumstances Mo set out, and providing 
some level of sub-frame resilience where possible in that case. If your 
assumption is that FUs are not being used, and re-writing is 
possible/desirable, you will certainly conclude that FUs should not be 
used. Since we concluded that FUs should not be discouraged, it seems 
reasonable to consider adding features which make them more resilient 
when they are.

best regards

Thomas

On 05/07/13 10:34, Thomas Schierl wrote:
> Am 04.07.2013 um 19:59 schrieb "Wang, Ye-Kui"<[email-masked]>:
>
>> Hi Thomas,
>>
>>>>> 6. If ERROR<  50%, it is likely all FUs are part of the same

---

From: Thomas Schierl <[email-masked]>
Date: Fri, 5 Jul 2013 16:38:55 +0200

Dear Thomas,

For the error resilience in the case of slices or dependent slices, which do match with tiles, I do not think an additional pointer information in the FU header does improve the error resilience capabilities over a system without those information. 

I think I did not exclude the use of FUs even for dependent slice fragments, but I would not require an additional mechanism to those, as stated in my email before.

Thomas
--

Dr.-Ing. Thomas Schierl

Head of Multimedia Communications Group
Image Processing Department
[email-masked]

Tel.:  +49 30 31002-227
Fax:  +49 30 31002-190
Skype:  thomas.schierl

Fraunhofer HHI - Heinrich Hertz Institute
Einsteinufer 37, 10587 Berlin, Germany
http://www.hhi.fraunhofer.de/ip/mc

Am 05.07.2013 um 16:20 schrieb Thomas Davies <[email-masked]>:

> Dear Thomas
> 
> On your first point, the decoder always has to be told there is missing data. I don't see much different or fancy in the interface between the two cases. Usually error concealment must be addressed both within the codec (because of reference dependencies) and also in the application (because of display and call-management/error response functionality). That's surely a very minor issue.
> 
> On the second point, the context we are considering in our proposal is where FUs are being used, in the circumstances Mo set out, and providing some level of sub-frame resilience where possible in that case. If your assumption is that FUs are not being used, and re-writing is possible/desirable, you will certainly conclude that FUs should not be used. Since we concluded that FUs should not be discouraged, it seems reasonable to consider adding features which make them more resilient when they are.
> 
> best regards
> 
> Thomas
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Fri, 5 Jul 2013 15:31:41 +0000

Hi Thomas,

Please see my reply to YK about why some RTP apps may prefer (or be forced to use) fragmentation rather than dependent slice segments. The current draft supports resilience using tiles with dependent slice segments. The proposed enhancement supports resilience using tiles with fragmentation.

I understand that dependent slice segments were specifically added to allow a form of codec level fragmentation at finer granularity than slices. However, they are not always sufficient, desirable or even available in all cases of interest for RTP apps. If they were, there would be no need for RTP level fragmentation at all. Since the current draft supports RTP level fragmentation, I assume there is consensus among the authors that there is value in it. The proposed enhancement extends that value by also supporting resilience, so RTP apps which use fragmentation can also support resilience.

Regards,
Mo

-----Original Message-----
From: Thomas Schierl [mailto:[email-masked]] 
Sent: Friday, July 05, 2013 10:39 AM
To: Thomas Davies (thdavies)
Cc: Wang, Ye-Kui; Mo Zanaty (mzanaty); Rickard Sjöberg; Stephan Wenger; Paul Bright-Thomas (paubrigh); [email-masked]
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

Dear Thomas,

Thomas
--

Dr.-Ing. Thomas Schierl

Tel.:  +49 30 31002-227
Fax:  +49 30 31002-190
Skype:  thomas.schierl

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 5 Jul 2013 15:57:09 +0000

Hi Mo,

FUs IMO are mainly used for fragmenting big slices for stored /pre-encoded contents. Though I agreed not to discourage the use of FUs for live-encoding scenarios, I agreed on the basis that MTU size matching can be used for other purposes than error resilience. For error resilience purposes in live-encoding scenarios, I still don't think using FUs is a good choice, as Stephan also mentioned and others basically agreed too. Note also that actually FUs came earlier (as they were included in H.264 payload and this draft inherited a lot of stuff from the H.264 payload spec) than dependent slices (only in H.265).

>From the discussions so far, for the error resilience optimization you guys are trying to achieve, my understanding is that using dependent slices can achieve at least the same performance, and conforming H.265 decoders are required to support dependent slices anyway.

BR, YK

> Sent: Friday, July 05, 2013 8:32 AM
> To: Thomas Schierl; Thomas Davies (thdavies)
> Cc: Wang, Ye-Kui; Rickard Sjöberg; Stephan Wenger; Paul Bright-Thomas
> (paubrigh); [email-masked]
> Please see my reply to YK about why some RTP apps may prefer (or be forced
> to use) fragmentation rather than dependent slice segments. The current draft
> supports resilience using tiles with dependent slice segments. The proposed
> enhancement supports resilience using tiles with fragmentation.
> 
> I understand that dependent slice segments were specifically added to allow a
> form of codec level fragmentation at finer granularity than slices. However,
> they are not always sufficient, desirable or even available in all cases of interest
> for RTP apps. If they were, there would be no need for RTP level fragmentation
> at all. Since the current draft supports RTP level fragmentation, I assume there
> is consensus among the authors that there is value in it. The proposed
> enhancement extends that value by also supporting resilience, so RTP apps
> which use fragmentation can also support resilience.
> 
> Regards,
> Mo
> 
> Cc: Wang, Ye-Kui; Mo Zanaty (mzanaty); Rickard Sjöberg; Stephan Wenger;
> Paul Bright-Thomas (paubrigh); [email-masked]
> For the error resilience in the case of slices or dependent slices, which do match
> with tiles, I do not think an additional pointer information in the FU header
> does improve the error resilience capabilities over a system without those
> information.
> 
> I think I did not exclude the use of FUs even for dependent slice fragments, but
> I would not require an additional mechanism to those, as stated in my email
> before.
> 
> Thomas
> --
> 
> Dr.-Ing. Thomas Schierl
> 
> Fraunhofer HHI - Heinrich Hertz Institute Einsteinufer 37, 10587 Berlin,
> Germany http://www.hhi.fraunhofer.de/ip/mc
> 
> > On your first point, the decoder always has to be told there is missing data. I
> don't see much different or fancy in the interface between the two cases.
> Usually error concealment must be addressed both within the codec (because
> of reference dependencies) and also in the application (because of display and
> call-management/error response functionality). That's surely a very minor issue.
> >
> > On the second point, the context we are considering in our proposal is where
> FUs are being used, in the circumstances Mo set out, and providing some level
> of sub-frame resilience where possible in that case. If your assumption is that
> FUs are not being used, and re-writing is possible/desirable, you will certainly
> conclude that FUs should not be used. Since we concluded that FUs should not
> be discouraged, it seems reasonable to consider adding features which make
> them more resilient when they are.
> >
> > best regards
> >
> > Thomas
> >
> >> Further I would tentatively assume that you can use the slice segment
> address in the dependent slice fragment header for the same degree of error
> resilience as the new-FU header, if you put a tile into a dependent slice. And
> this part would be understood by the decoder itself, which should be able to do
> the error resilience on codec level. Without fancy interfaces into the decoder,
> etc.
> >>
> >> A point in the discussion which seems to be missing is that the rewriting of a
> single slice into multiple dependent slice fragments (including also one
> independent slice fragment) can be done at very low processing costs, as shown
> during the standardization of HEVC.
> >>
> >> Thomas
> >> --
> >> Thomas Schierl
> >> Fraunhofer HHI
> >>
> >>> So you did not intent to encapsulate only entire tiles into packets, though
> that would obviously be more error resilient as you won't encounter the
> problem where a part of a tile gets lost and the received rest becomes useless.
> Rather, one tile may be carried in two (or even more) packets.
> >>>
> >>> I think I was confused by the suggested language such as "If a
> fragmentation unit is lost, tiles in the lost fragmentation unit are also lost.
> despite lost fragmentation units.", from where I got an impression that an FU
> includes entire tiles.
> >>>
> >>> As seen from other discussions, it is very questionable how big a penalty
> would be when an encoder tries to splitting encoded data into separate NAL
> units (slices or dependent slices) for MTU size matching, even for large CTB
> sizes - that's the reason why the JCT-VC later decided to remove the fine-
> granularity slice feature.
> >>>
> >>> Let's tentatively assume that overall penalty and the overhead of the new-
> FU approach are roughly equivalent, and take into account latest comments
> from Mo and Stephan, do we see an advantage of the new-FU approach
> compared to using dependent slices?
> >>>
> >>> I am not trying to conclude yet - just try to (very briefly) summarize the
> situation so far at least for myself.  We should certainly have more discussions -
> preferable involving more people who care about this payload format.
> >>>
> >>> BR, YK
> >>>
> >>>> With the proposed scheme there is no need for the RTP packetizer to
> >>>> parse slice headers or identify tile boundaries. All it needs to do
> >>>> is to keep track of the offset of the start of the payload to the
> >>>> beginning of the NALU and put this into the field. It needs to know
> >>>> this offset anyway. The packetizer is completely dumb.
> >>>>
> >>>> The point is that a decoder wishing to do error resilience can
> >>>> parse the slice header and identify the tile boundaries in the case
> >>>> that a packet goes missing, for tiles that occur in the same frame
> >>>> but after the packet that is lost. The decoder can compare the tile
> >>>> offsets with the FU offsets to work out what data is missing.
> >>>>
> >>>> Re dependent slices, there are indirect overheads as well, which
> >>>> come from having to quantize the packet size to a whole number of
> >>>> tiles, plus you have to manage the encoder to match the MTU size.
> >>>> The granularity of tiles you need for your parallel processing may
> >>>> not match the granularity you need to fit in the MTU very well.
> >>>>
> >>>> Best regards
> >>>>
> >>>> Thomas
> >>>>
> >>>>
> >>>> To: Thomas Davies (thdavies); Mo Zanaty (mzanaty); Rickard Sjöberg;
> >>>> My point was really about using dependent slices, not about 1 or N
> >>>> tiles per packets (sorry that I did not make that clearer).
> >>>> Whatever strategy you would like to do in terms of the number of
> >>>> tiles in a packet, using dependent slices can do the same, as each
> dependent slice can include 1 or N tiles.
> >>>>
> >>>> - The RTP packetizer does not need to parse slice headers to
> >>>> identify boundaries of tiles within a coded slice, as tiles of one
> >>>> SLICE are already in separate NAL units each containing a DEPENDENT
> SLICE.
> >>>>
> >>>> - No additional overhead in packetization (the 1-byte FU header and
> >>>> the 2- or 4-byte FU offset). The overhead of the short slice header
> >>>> for dependent slice NAL units is compensated by less number of tile
> >>>> entry offsets signalled, as the entry offset of the first tile in each NAL unit
> is not signalled in the slice header.
> >>>> Using dependent slices would need more two-byte NAL unit headers,
> >>>> while using new FUs would need more two-byte packet payload
> >>>> headers, and the overhead bits here are the same for both methods.
> >>>>
> >>>>> The issue is that you need to guarantee that you are doing this
> >>>>> for it to be useful, and the decoder needs to know that this is
> >>>>> guaranteed through some form of other signalling. However, the
> >>>>> bitstream chunk associated to a tile is extremely variable, and
> >>>>> many tile chunks would be very small. So there is significant
> >>>>> packetization overhead for them if there
> >>>> is one tile per packet.
> >>>>> especially if you are already using them for parallelization. Thus
> >>>>> you might have to break the guarantee and fragment the packet
> >>>>> anyhow, losing
> >>>> the resilience.
> >>>>> The advantage that we see with the proposal is that you can get
> >>>>> fairly similar resilience to regular slice MTU-size matching, but
> >>>>> with much lower overheads as you can have a single regular slice
> >>>>> header per frame, yet
> >>>> have a "dumb"
> >>>>> In H265 it involves re-encoding at least one 64x64 CTB, and if you
> >>>>> are doing tiles then you can get tiny "dangling slices" at the end
> >>>>> of a tile, which means more small packets.
> >>>>>
> >>>>> Best regards
> >>>>>
> >>>>> Thomas
> >>>>>
> >>>>> To: Mo Zanaty (mzanaty); Rickard Sjöberg; Thomas Davies
> >>>>> (thdavies); Stephan Wenger; Paul Bright-Thomas (paubrigh)
> >>>>> Cc: [email-masked]
> >>>>> An interesting idea. Using dependent slices (again :-) together
> >>>>> with tiles would do the trick too (each tile in one dependent
> >>>>> slice, and then each dependent slice NAL unit in one single NAL unit
> packet).
> >>>>>
> >>>>>> To: Rickard Sjöberg; Wang, Ye-Kui; Thomas Davies (thdavies);
> >>>>>> resilience using fragments and tiles. I mentioned this to Stephan
> >>>>>> at IETF 86 in Orlando, and promised to have draft text ready
> >>>>>> before Berlin. Here is the proposed text along with an overview of the
> rationale.
> >>>>>>
> >>>>>> Rationale:
> >>>>>>
> >>>>>> Since independence requires restrictions on some coding tools,
> >>>>>> both potentially incur slight penalties in coding efficiency,
> >>>>>> with a smaller penalty
> >>>>> for tiles.
> >>>>>> a. Efficiency: RTP fragmentation has finer (byte) granularity
> >>>>>> while slices have much larger (CTB or tile) granularity. It also
> >>>>>> has less slice segment header overhead, especially for independent
> slices.
> >>>>>> This can impact the efficiency of the packetized stream in terms
> >>>>>> of both bits and
> >>>>> packets per second.
> >>>>>> statistics (which aggravates a.), while others recode a slice if
> >>>>>> the size is exceeded (which
> >>>>> aggravates b.).
> size.
> >>>>>> retaining the above advantages, if fragments are enhanced to
> >>>>>> contain offsets (similar to IP fragmentation). This is described
> >>>>>> in the proposed draft text below, which would immediately follow
> >>>>>> section
> >>>>>> 4.7
> >>>>> Fragmentation Units.
> >>>>>> Draft Text:
> >>>>>>
> >>>>>> contains offsets to the data of each tile. These offsets can be
> >>>>>> used for random access to any tile for parallel processing,
> >>>>>> region of interest extraction, and resilience to errors in other
> >>>>>> tiles. These offsets can also be used to provide resilience to
> >>>>>> packet loss, in conjunction with fragment offsets contained in
> >>>>>> two types of fragmentation
> >>>>> units: FU-A2 and FU-A4.
> >>>>>> Fragmentation units FU-A2 and FU-A4 contain a fragment offset
> >>>>>> before the FU payload. FU-A2 contains a 2-byte offset as shown in
> Figure X.
> >>>>>> indicates the total number of bytes in all prior FU payloads of
> >>>>>> the fragmented
> >>>>> NAL unit.
> >>>>>>
> >>>>>>    0                   1                   2                   3
> >>>>>> Since the offset of the first FU payload of a fragmented NAL unit
> >>>>>> is zero, and the Start (S) bit in the FU header is sufficient to
> >>>>>> indicate this, the first fragmentation unit of a fragmented NAL
> >>>>>> unit SHOULD use FU-A, but MAY use
> >>>>>> If a fragmentation unit is lost, tiles in the lost fragmentation
> >>>>>> unit are also
> >>>> lost.
> >>>>>> However, tiles which are successfully received in their entirety
> >>>>>> can be decoded if their slice segment header containing tile
> >>>>>> offsets is successfully received, despite lost fragmentation
> >>>>>> units. Fragment offsets in FU-A2 or FU-A4 are required in order
> >>>>>> to recover from loss of prior fragmentation units and continue decoding
> subsequent tiles.
> >>>>>>
> >>>>>> The following heuristic SHOULD be applied to determine if the
> >>>>>> lost fragmentation units are all part of the same fragmented NAL
> >>>>>> unit as subsequently received fragmentation units with identical
> >>>>>> RTP timestamps and identical values of NAL unit header layer ID
> >>>>>> and temporal
> >>>> ID.
> >>>>>> On Behalf Of Rickard Sjöberg
> >>>>>> I wrote slices and not dependent slices since the slice
> >>>>>> granularity and slice/tile rule applies to both slices and dependent slices.
> >>>>>> I believe that FUs are a viable option to dependent slices for
> >>>>>> the reasons I listed in my previous e-mail and think that we
> >>>>>> should not recommend dependent slices over FUs but leave it to the
> implementer.
> >>>>>>
> >>>>>> BR
> >>>>>> Rickard
> >>>>>>
> >>>>>>
> >>>>>> I drafted an email replying directly to Rickard but did not
> >>>>>> finish and there was a meeting (and then Stephan's and Thomas' emails
> came).
> >>>>>>
> >>>>>> slices, not FUs vs dependent slices, while the recommendation in
> >>>>>> the draft is recommending use of dependent slices over FUs in
> >>>>>> For live encoding, whenever splitting of a regular slice is
> >>>>>> needed, the splitting can be done by the encoder using dependent
> >>>>>> slices - and the encoder has more knowledge and is more powerful
> >>>>>> than the RTP packetizer and thus can do a better job for the splitting.
> >>>>>>
> >>>>>> BR, YK
> >>>>>>
> >>>>>>
> >>>>>>> In general FUs may be used where error conditions are known to
> >>>>>>> be benign, for greater efficiency, and also where for whatever
> >>>>>>> reason the encoder cannot support MTU matching.
> >>>>>>>
> >>>>>>>> and you can recover at the next slice boundary; loose an FU,
> >>>>>>>> and you have to wait for the next slice header and throw away
> >>>>>>>> all trailing FUs.  RTP is in virtually all cases run over UDP,
> >>>>>>>> and in the typical Internet scenario UDP is lossy (in contrast
> >>>>>>>> to private, managed IP-network, which may have other
> >>>>>>>> constraints, but are not the IETF's main
> >>>>>>>> needs to be inserted.  You may remember a quick analysis of
> >>>>>>>> mine we discussed way back early in JCT, where we (I think)
> >>>>>>>> agreed that the practical impact is negligible in almost all
> >>>>>>>> non-tile scenarios (mostly because an additional overhead of 50
> >>>>>>>> bytes or so--3% or less of the coded picture size) is incurred
> >>>>>>>> implementations for relatively small picture sizes some folks
> >>>>>>>> have been
> >>>> considering.
> >>>>>>>> OTOH, because FUs are trivial to implement--some say in
> >>>>>>>> contrast to slices--and because (I hope) we all agree that
> >>>>>>>> using FUs in error prone scenarios is a bad idea if you could
> >>>>>>>> use slices (but for reasons like the tile connection you
> >>>>>>>> mentioned), the draft should IMO continue to set a preference
> >>>>>>>> for the use of regular slices over FUs.  To avoid
> >>>>>>>>> aggregation. The spatial granularity of slices was 16x16
> >>>>>>>>> pixels in H.264 but is typically
> >>>>>>>>> easier to implement since you do not need to care about when
> >>>>>>>>> to break your slice to not exceed the MTU size limit. I think
> >>>>>>>>> both slices and FUs has their merits and the choice between
> >>>>>>>>> them for live-encoding should be left for
> >>>>>> the implementer.
> >>>>>>>>> Regarding the use of PicOrderCntVal for feedback in 8.2 and
> >>>>>>>>> 8.2 and the POC MSB sync issue, I agree that this is a corner
> >>>>>>>>> case that we probably will not see much of in practice. I have
> >>>>>>>>> no text change suggestion at the moment.
> >>>>>>>>>
> >>>>>>>>> BR
> >>>>>>>>> Rickard
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks for the careful review and the comments. Please see
> >>>>>>>>> inline
> >>>> below.
> >>>>>>>>> BR, YK
> >>>>>>>>>
> >>>>>>>>>> I have taken a first look at
> >>>>>>>>>> draft-schierl-payload-rtp-h265-03 and have some
> questions/comments.
> >>>>>>>>>>
> >>>>>>>>> sort of copy-and-paste error. Thanks for the good catching.
> >>>>>>>>> No, there is no intention to cover HEVC extensions in this
> >>>>>>>>> payload specification, though we intentionally to have the
> >>>>>>>>>> pass picture timing SEI messages and decoding unit
> >>>>>>>>>> information SEI messages to the decoder"?
> >>>>>>>>> checking for the bitstream that was possible would be
> >>>>>>>>> disabled, and also the information related to frame
> >>>>>>>>> to put this as a packetization rule is to put it as part of
> >>>>>>>>> the semantics of NAL unit header (1.1.4), but that sub-section
> >>>>>>>>> live-encoding scenarios, wherein dependent slice segments
> >>>>>>>>> should be
> >>>> used instead.
> >>>>>>>>> need of whether to specify a payload structure for mixed FU
> >>>>>>>>> and aggregation packets and from that discussion we concluded
> >>>>>>>>> that encoding dependent slice segments at source coding level
> >>>>>>>>> already allows for the use cases and using of FUs for
> >>>>>>>>> live-encoding does not make sense. Please let us know if you
> >>>>>>>>> think differently.
> >>>>>>>>>
> >>>>>>>>> random accessing to a bitstream is performed. However, it
> >>>>>>>>> would not happen in conversational applications where the
> >>>>>>>>> feedback messages may
> >>>>> be used.
> >>>>>>>>>> "2) An indication of whether there is any parameter set
> >>>>>>>>>> within the current coded video sequence that updates another
> >>>>>>>>>> http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h2
> >>>>>>>>>> 6
> >>>>>>>>>> 5-
> >>>>>>>>>> 02
> >>>>>>>>>>
> >>>>>>>>>> Version 03:
> >>>>>>>>>> URL:
> >>>>>>>>>> http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h2
> >>>>>>>>>> 6
> >>>>>>>>>> 5-
> >>>>>>>>>> 03
> >>>>>>>>>>
> >>>>>>>>>> other standards bodies such as 3GPP rely on the payload
> >>>>>>>>>> format for support of H.265/HEVC in RTP based video services,
> >>>>>>>>>> we need to progress this draft relatively quickly.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 8 Jul 2013 03:48:33 +0000

Hi YK,

This is not about decoder conformance. Almost all decoders fully conform to the standards they support (for their advertised profiles/levels). This is about encoder feature set and APIs for tool selection. Even today, after a decade of H.264, most popular hardware chipsets don't support encoder MTU slicing at all, or with degraded performance, or with constraints on other features (e.g. CABAC). Preliminary information on H.265 plans seems to mirror this.

While I certainly agree we should influence encoders to support dependent slices, I think we should also give RTP applications the best tools to work with any encoder (or at least the most popular encoders). Remember, this is the spec for RTP applications, not the spec for encoder designers.

Cheers,
Mo

-----Original Message-----
From: Wang, Ye-Kui [mailto:[email-masked]] 
Sent: Friday, July 05, 2013 11:57 AM
To: Mo Zanaty (mzanaty); Thomas Schierl; Thomas Davies (thdavies)
Cc: Rickard Sjöberg; Stephan Wenger; Paul Bright-Thomas (paubrigh); [email-masked]
Subject: RE: [payload] Submission of new versions of H.265/HEVC payload format

Hi Mo,

BR, YK

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Mon, 8 Jul 2013 06:35:12 +0000

Hi Mo,

Implementing dependent slices in either encoder side or decoder side IMO is trivial.

In all those standards organizations that I have ever worked, including here, the experience that I have learned is that any new tool, particularly a normative standard feature, needs to provide good improvements against methods using existing tools, to be justified to be added to a specification, as adding new tools always incur additional costs in specifying, implementation, testing, and so on.

Arguments like "This is about the RTP payload, thus we can ignore some coding tools specified in the coding specification" are not convincing to me. In video coding development, people would often need to discuss tools that are available in systems, including in RTP environments, when trying to judging whether a coding tool proposal makes sense. In different standards bodies, features addressing some issues because of non-desirable implementations are often seen and are more often seen not agreed.

BR, YK

> Sent: Sunday, July 07, 2013 8:49 PM
> To: Wang, Ye-Kui; Thomas Schierl; Thomas Davies (thdavies)
> This is not about decoder conformance. Almost all decoders fully conform to
> the standards they support (for their advertised profiles/levels). This is about
> encoder feature set and APIs for tool selection. Even today, after a decade of
> H.264, most popular hardware chipsets don't support encoder MTU slicing at
> all, or with degraded performance, or with constraints on other features (e.g.
> CABAC). Preliminary information on H.265 plans seems to mirror this.
> 
> While I certainly agree we should influence encoders to support dependent
> slices, I think we should also give RTP applications the best tools to work with
> any encoder (or at least the most popular encoders). Remember, this is the
> spec for RTP applications, not the spec for encoder designers.
> 
> Cheers,
> Mo
> 
> FUs IMO are mainly used for fragmenting big slices for stored /pre-encoded
> contents. Though I agreed not to discourage the use of FUs for live-encoding
> scenarios, I agreed on the basis that MTU size matching can be used for other
> purposes than error resilience. For error resilience purposes in live-encoding
> scenarios, I still don't think using FUs is a good choice, as Stephan also
> mentioned and others basically agreed too. Note also that actually FUs came
> earlier (as they were included in H.264 payload and this draft inherited a lot of
> stuff from the H.264 payload spec) than dependent slices (only in H.265).
> 
> From the discussions so far, for the error resilience optimization you guys are
> trying to achieve, my understanding is that using dependent slices can achieve
> at least the same performance, and conforming H.265 decoders are required to
> support dependent slices anyway.
> 
> BR, YK
> 
> > Please see my reply to YK about why some RTP apps may prefer (or be
> > forced to use) fragmentation rather than dependent slice segments. The
> > current draft supports resilience using tiles with dependent slice
> > segments. The proposed enhancement supports resilience using tiles with
> fragmentation.
> >
> > I understand that dependent slice segments were specifically added to
> > allow a form of codec level fragmentation at finer granularity than
> > slices. However, they are not always sufficient, desirable or even
> > available in all cases of interest for RTP apps. If they were, there
> > would be no need for RTP level fragmentation at all. Since the current
> > draft supports RTP level fragmentation, I assume there is consensus
> > among the authors that there is value in it. The proposed enhancement
> > extends that value by also supporting resilience, so RTP apps which use
> fragmentation can also support resilience.
> >
> > Regards,
> > Mo
> >
> > Cc: Wang, Ye-Kui; Mo Zanaty (mzanaty); Rickard Sjöberg; Stephan
> > Wenger; Paul Bright-Thomas (paubrigh); [email-masked]
> > For the error resilience in the case of slices or dependent slices,
> > which do match with tiles, I do not think an additional pointer
> > information in the FU header does improve the error resilience
> > capabilities over a system without those information.
> >
> > I think I did not exclude the use of FUs even for dependent slice
> > fragments, but I would not require an additional mechanism to those,
> > as stated in my email before.
> >
> > Thomas
> > --
> >
> > Dr.-Ing. Thomas Schierl
> >
> > Head of Multimedia Communications Group Image Processing Department
> > Fraunhofer HHI - Heinrich Hertz Institute Einsteinufer 37, 10587
> > Berlin, Germany http://www.hhi.fraunhofer.de/ip/mc
> >
> > Usually error concealment must be addressed both within the codec
> > (because of reference dependencies) and also in the application
> > (because of display and call-management/error response functionality). That's
> surely a very minor issue.
> > >
> > FUs are being used, in the circumstances Mo set out, and providing
> > some level of sub-frame resilience where possible in that case. If
> > your assumption is that FUs are not being used, and re-writing is
> > reasonable to consider adding features which make them more resilient when
> they are.
> > >
> > > best regards
> > >
> > > Thomas
> > >
> > address in the dependent slice fragment header for the same degree of
> > error resilience as the new-FU header, if you put a tile into a
> > dependent slice. And this part would be understood by the decoder
> > itself, which should be able to do the error resilience on codec
> > level. Without fancy interfaces into the decoder, etc.
> > >>
> > single slice into multiple dependent slice fragments (including also
> > one independent slice fragment) can be done at very low processing
> > costs, as shown during the standardization of HEVC.
> > >>
> > >> Thomas
> > >> --
> > >> Thomas Schierl
> > >> Fraunhofer HHI
> > >>
> > that would obviously be more error resilient as you won't encounter
> > the problem where a part of a tile gets lost and the received rest becomes
> useless.
> > successfully received, despite lost fragmentation units.", from where
> > I got an impression that an FU includes entire tiles.
> > >>>
> > would be when an encoder tries to splitting encoded data into separate
> > NAL units (slices or dependent slices) for MTU size matching, even for
> > large CTB sizes - that's the reason why the JCT-VC later decided to
> > remove the fine- granularity slice feature.
> > >>>
> > FU approach are roughly equivalent, and take into account latest
> > comments from Mo and Stephan, do we see an advantage of the new-FU
> > approach compared to using dependent slices?
> > >>>
> > situation so far at least for myself.  We should certainly have more
> > discussions - preferable involving more people who care about this payload
> format.
> > >>>
> > >>> BR, YK
> > >>>
> > >>>> With the proposed scheme there is no need for the RTP packetizer
> > >>>> to parse slice headers or identify tile boundaries. All it needs
> > >>>> to do is to keep track of the offset of the start of the payload
> > >>>> to the beginning of the NALU and put this into the field. It
> > >>>> needs to know this offset anyway. The packetizer is completely dumb.
> > >>>>
> > >>>> parse the slice header and identify the tile boundaries in the
> > >>>> case that a packet goes missing, for tiles that occur in the same
> > >>>> frame but after the packet that is lost. The decoder can compare
> > >>>> the tile offsets with the FU offsets to work out what data is missing.
> > >>>>
> > >>>> The granularity of tiles you need for your parallel processing
> > >>>> may not match the granularity you need to fit in the MTU very well.
> > >>>>
> > >>>> Best regards
> > >>>>
> > >>>> Thomas
> > >>>>
> > >>>>
> > >>>> To: Thomas Davies (thdavies); Mo Zanaty (mzanaty); Rickard
> > >>>> Sjöberg; Stephan Wenger; Paul Bright-Thomas (paubrigh)
> > >>>> Cc: [email-masked]
> > >>>> My point was really about using dependent slices, not about 1 or
> > >>>> N tiles per packets (sorry that I did not make that clearer).
> > >>>> identify boundaries of tiles within a coded slice, as tiles of
> > >>>> one SLICE are already in separate NAL units each containing a
> > >>>> DEPENDENT
> > SLICE.
> > >>>>
> > >>>> - No additional overhead in packetization (the 1-byte FU header
> > >>>> and the 2- or 4-byte FU offset). The overhead of the short slice
> > >>>> header for dependent slice NAL units is compensated by less
> > >>>> number of tile entry offsets signalled, as the entry offset of
> > >>>> the first tile in each NAL unit
> > >>>>> also: I guess that is where we started from. Then you could
> > >>>>> indeed align missing tiles with missing sequence numbers.
> > >>>>>
> > >>>>> requires that you have to re- encode to fit things in and meet
> > >>>>> the guarantee. There is definitely no time to re-encode whole
> > >>>>> tiles, especially if you are already using them for
> > >>>>> parallelization. Thus you might have to break the guarantee and
> > >>>>> fragment the packet anyhow, losing
> > >>>> the resilience.
> > >>>>> fairly similar resilience to regular slice MTU-size matching,
> > >>>>> but with much lower overheads as you can have a single regular
> > >>>>> slice header per frame, yet
> > >>>> have a "dumb"
> > >>>>> H264 MTU matching usually means re-encoding just one or two
> > >>>>> 16x16
> > MBs.
> > >>>>> In H265 it involves re-encoding at least one 64x64 CTB, and if
> > >>>>> you are doing tiles then you can get tiny "dangling slices" at
> > >>>>> the end of a tile, which means more small packets.
> > >>>>>
> > >>>>> Best regards
> > >>>>>
> > >>>>> Thomas
> > >>>>>
> > >>>>>> resilience using fragments and tiles. I mentioned this to
> > >>>>>> Stephan at IETF 86 in Orlando, and promised to have draft text
> > >>>>>> ready before Berlin. Here is the proposed text along with an
> > >>>>>> overview of the
> > rationale.
> > >>>>>>
> > >>>>>> Rationale:
> > >>>>>>
> > >>>>>> while slices have much larger (CTB or tile) granularity. It
> > >>>>>> also has less slice segment header overhead, especially for
> > >>>>>> independent
> > slices.
> > >>>>>> This can impact the efficiency of the packetized stream in
> > >>>>>> terms of both bits and
> > >>>>> packets per second.
> > >>>>>> statistics (which aggravates a.), while others recode a slice
> > >>>>>> if the size is exceeded (which
> > >>>>> aggravates b.).
> > >>>>>> Applications (for live or stored content) using such encoders
> > >>>>>> can still support MTU limits using RTP fragmentation since it
> > >>>>>> is decoupled from
> > >>>>> the encoder.
> > >>>>>> d. Multi-party: The encoded content may go to multiple
> > >>>>>> receivers with different MTUs. RTP fragmentation can handle
> > >>>>>> this with lightweight repacketization at middle boxes whereas
> > >>>>>> slices incur
> > >>>> heavyweight transcoding.
> > >>>>>> 3. Combining fragmentation and tiles can improve resilience
> > >>>>>> while retaining the above advantages, if fragments are enhanced
> > >>>>>> to contain offsets (similar to IP fragmentation). This is
> > >>>>>> described in the proposed draft text below, which would
> > >>>>>> immediately follow section
> > >>>>>> 4.7
> > >>>>> Fragmentation Units.
> > >>>>>> Draft Text:
> > >>>>>>
> > >>>>>> If a fragmented NAL unit contains tiles, its slice segment
> > >>>>>> header contains offsets to the data of each tile. These offsets
> > >>>>>> can be used for random access to any tile for parallel
> > >>>>>> processing, region of interest extraction, and resilience to
> > >>>>>> errors in other tiles. These offsets can also be used to
> > >>>>>> provide resilience to packet loss, in conjunction with fragment
> > >>>>>> offsets contained in two types of fragmentation
> > >>>>> units: FU-A2 and FU-A4.
> > >>>>>> Since the offset of the first FU payload of a fragmented NAL
> > >>>>>> unit is zero, and the Start (S) bit in the FU header is
> > >>>>>> sufficient to indicate this, the first fragmentation unit of a
> > >>>>>> fragmented NAL unit SHOULD use FU-A, but MAY use
> > >>>>>> If a fragmentation unit is lost, tiles in the lost
> > >>>>>> fragmentation unit are also
> > >>>> lost.
> > >>>>>> However, tiles which are successfully received in their
> > >>>>>> entirety can be decoded if their slice segment header
> > >>>>>> containing tile offsets is successfully received, despite lost
> > >>>>>> fragmentation units. Fragment offsets in FU-A2 or FU-A4 are
> > >>>>>> required in order to recover from loss of prior fragmentation
> > >>>>>> units and continue decoding
> > subsequent tiles.
> > >>>>>>
> > >>>>>> lost fragmentation units are all part of the same fragmented
> > >>>>>> NAL unit as subsequently received fragmentation units with
> > >>>>>> identical RTP timestamps and identical values of NAL unit
> > >>>>>> header layer ID and temporal
> > >>>> ID.
> > >>>>>> [mailto:[email-masked]] On Behalf Of Rickard Sjöberg
> > >>>>>> Sorry for the confusion. I will now use 'independent slices'
> > >>>>>> and 'dependent slices' to distinguish between them.
> > >>>>>>
> > >>>>>> I wanted to comment that Rickard was comparing the use of FUs
> > >>>>>> vs slices, not FUs vs dependent slices, while the
> > >>>>>> recommendation in the draft is recommending use of dependent
> > >>>>>> slices over FUs in live-encoding scenarios (so this answers
> > >>>>>> Thomas question: the intention was
> > >>>>> dependent slices ).
> > >>>>>> In any case, it does not make much sense to try MTU size
> > >>>>>> matching and at the same time use either dependent slices or
> > >>>>>> FUs, because dependent slices and FUs should not be used when
> > >>>>>> there are considerable packet losses, while MTU size matching
> > >>>>>> should be applied when there are
> > >>>>>> needed, the splitting can be done by the encoder using
> > >>>>>> dependent slices - and the encoder has more knowledge and is
> > >>>>>> more powerful than the RTP packetizer and thus can do a better job
> for the splitting.
> > >>>>>>
> > >>>>>> BR, YK
> > >>>>>>
> > >>>>>>
> > >>>>>>> [mailto:[email-masked]] On Behalf Of Thomas Davies
> > >>>>>>> I think the text in question states a preference for
> > >>>>>>> _dependent_ slice segments, which have no error resiliency
> advantage over FUs.
> > >>>>>>> Is this just a
> > >>>>>> typo?
> > >>>>>>> <snip>
> > >>>>>>>
> > >>>>>>>       broadcast, in which cases dependent slice segments
> > >>>>>>> SHOULD be
> > used
> > >>>>>>> We also are formulating some more general comments on this
> > >>>>>>> area which we hope to provide more fully soon.
> > >>>>>>>
> > >>>>>>> best regards
> > >>>>>>>
> > >>>>>>> Thomas
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> The preference for slices over FUs, historically, was pushed
> > >>>>>>>> by myself into RFC 3984 for reasons of error resilience, and
> > >>>>>>>> was moved over to this draft for the same reason.  Loose a
> > >>>>>>>> slice, and you can recover at the next slice boundary; loose
> > >>>>>>>> an FU, and you have to wait for the next slice header and
> > >>>>>>>> throw away all trailing FUs.  RTP is in virtually all cases
> > >>>>>>>> run over UDP, and in the typical Internet scenario UDP is
> > >>>>>>>> lossy (in contrast to private, managed IP-network, which may
> > >>>>>>>> have other constraints, but are not the IETF's main
> > >>>>>>>> implementation complexity, but we should not be in the
> > >>>>>>>> business to encourage implementor
> > >>>>>> laziness when
> > >>>>>>>> result of incompetently filled packets (which, I agree, is
> > >>>>>>>> more likely with
> > >>>>>>>> non-tile scenarios (mostly because an additional overhead of
> > >>>>>>>> 50 bytes or so--3% or less of the coded picture size) is
> > >>>>>>>> incurred statistically only rarely (only in those cases where
> > >>>>>>>> an additional packet would need to be generated because of
> > >>>>>>>> the video coding related overhead for the use of slices).
> > >>>>>>>> The prediction overhead of the use of regular slices has been
> > >>>>>>>> shown to be substantial--the price one needs to pay for
> > >>>>>>>> independent decodability--but for entropy slices that
> > >>>>>>>> overhead virtually
> > >>>>>> does not exist.
> > >>>>>>>> express a preference of the use of regular slices (SHOULD)
> > >>>>>>>> when you expect losses and can use them (real-time encoding,
> > >>>>>>>> no tiling structure that would lead to unacceptable
> > >>>>>>>> packetization overhead); and suggest (MAY) the use of FUs in other
> scenarios.
> > >>>>>>>>> Thanks for your feedback on my comments, your suggestions
> > >>>>>>>>> look good to
> > >>>>>>> me.
> > >>>>>>>>> live-encoding scenarios in section 4.7, I think using FUs
> > >>>>>>>>> does make sense when you do not have many non-VLC NAL units
> > >>>>>>>>> for aggregation. The spatial granularity of slices was 16x16
> > >>>>>>>>> 64x64 for HEVC which means that MTU size matching is done
> > >>>>>>>>> with units that are 16x larger. This may lead to a larger
> > >>>>>>>>> number of smaller packets when slices are used compared to
> > >>>>>>>>> FUs. In addition, there is the HEVC rule of slice and tile
> > >>>>>>>>> boundaries that makes the slice granularity equal to the
> > >>>>>>>>> tile granularity for cases when slices span multiple tiles.
> > >>>>>>>>> Finally, FUs are easier to implement since you do not need
> > >>>>>>>>> to care about when to break your slice to not exceed the MTU
> > >>>>>>>>> size limit. I think both slices and FUs has their merits and
> > >>>>>>>>> the choice between them for live-encoding should be left for
> > >>>>>> the implementer.
> > >>>>>>>>> 8.2 and the POC MSB sync issue, I agree that this is a
> > >>>>>>>>> corner case that we probably will not see much of in
> > >>>>>>>>> practice. I have no text change suggestion at the moment.
> > >>>>>>>>>
> > >>>>>>>>> BR
> > >>>>>>>>> Rickard
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> [YK] That is in the semantics of LayerId (nuh_layer_id),
> > >>>>>>>>> where "this specification" should be replaced with "[HEVC]".
> > >>>>>>>>> It is sort of copy-and-paste error. Thanks for the good catching.
> > >>>>>>>>> depacketization process and the use with feedback messages
> > >>>>>>>>> to work for HEVC scalability
> > >>>>>> and 3DV extensions.
> > >>>>>>>>>> Another question for my understanding is why "Receivers
> > >>>>>>>>>> MUST pass picture timing SEI messages and decoding unit
> > >>>>>>>>> specified by the video coding spec to the video decoders.
> > >>>>>>>>> This is emphasized here for the two timing related SEI
> > >>>>>>>>> messages after it is said that picture output timing in RTP
> > >>>>>>>>> timestamps should be used instead (to avoid giving a wrong
> > >>>>>>>>> impression to readers that the SEI messages should be
> > >>>>>>>>> ignored). If an RTP receiver discards the SEI messages, then
> > >>>>>>>>> HRD conformance checking for the bitstream that was possible
> > >>>>>>>>> would be disabled, and also the information related to frame
> > >>>>>>>>> [YK] I see your point, it is only a "MAY" rule. An
> > >>>>>>>>> alternative to put this as a packetization rule is to put it
> > >>>>>>>>> as part of the semantics of NAL unit header (1.1.4), but
> > >>>>>>>>> that sub-section belongs to an overview wherein normative
> > >>>>>>>>> language (even "MAY") should not appear. One solution to
> > >>>>>>>>> this is to insert "Payload Header
> > >>>>> Usage"
> > >>>>>>>>> and aggregation packets and from that discussion we
> > >>>>>>>>> concluded that encoding dependent slice segments at source
> > >>>>>>>>> coding level already allows for the use cases and using of
> > >>>>>>>>> FUs for live-encoding does not make sense. Please let us
> > >>>>>>>>> know if you think differently.
> > >>>>>>>>>
> > >>>>>>>>>> The use of spatial-segmentation-idc may need some updates
> > >>>>>>>>>> to be useful, let me come back to that later on.
> > >>>>>>>>>>
> > >>>>>>>>>> Replace "In addition, interpolation filter" by "In
> > >>>>>>>>>> addition, the interpolation filter"
> > >>>>>>>>> [YK] Thanks - done.
> > >>>>>>>>>
> > >>>>>>>>>> within the current coded video sequence that updates
> > >>>>>>>>>> "2) An indication of whether there is no parameter set
> > >>>>>>>>>> update in the coded video sequence."
> > >>>>>>>>> [YK] Changed the wording to "An indication of whether there
> > >>>>>>>>> is no parameter set within the current coded video sequence
> > >>>>>>>>> that updates another parameter set of the same type
> > >>>>>>>>> preceding in decoding
> > >>>>> order."
> > >>>>>>>>>> Section 4.7:
> > >>>>>>>>>> Type:  6 bits" to avoid confusion with the Type in the
> > >>>>>>>>>> payload
> > header.
> > >>>>>>>>>>
> > >>>>>>>>> [YK] We are consistently using "id" for profile-id,
> > >>>>>>>>> level-id, max-recv-level-id etc, similarly as in RFC 6184, instead of
> "idc".
> > >>>>>>>>>
> > >>>>>>>>>> Seems to me that "There MUST be exactly one RPSI contained
> > >>>>>>>>>> in the FCI field" should be "There MUST be exactly one SPLI
> > >>>>>>>>>> http://www.ietf.org/internet-drafts/draft-schierl-payload-r
> > >>>>>>>>>> tp
> > >>>>>>>>>> -
> > >>>>>>>>>> h265-02.txt
> > >>>>>>>>>> Htmlized:
> > >>>>>>>>>> http://www.ietf.org/internet-drafts/draft-schierl-payload-r
> > >>>>>>>>>> tp
> > >>>>>>>>>> -
> > >>>>>>>>>> h265-03.txt
> > >>>>>>>>>> Htmlized:
> > >>>>>>>>>> worked hard to try to make the design complete, from both
> > >>>>>>>>>> the payload structure and the signaling points of view.
> > >>>>>>>>>> Compared to version 02, version
> > >>>>>>>>>> format for support of H.265/HEVC in RTP based video
> > >>>>>>>>>> services, we need to progress this draft relatively quickly.

---

From: Thomas Schierl <[email-masked]>
Date: Mon, 8 Jul 2013 12:10:42 +0200

Dear Mo,

I think I cannot  agree with your point "they are not always sufficient, desirable or even available in all cases of interest for RTP apps. " If you are using the codec as it is supposed to be used, they should be sufficient as presented in my earlier email and they should be available. I am not sure about the term "desirable" wrt. standards.

However, if there should be a very simple application, which wants to make use of the advanced codec features of HEVC without having the codec itself under control, I may be able to agree in using FUs but only under the prerequisite that in the normal case you SHOULD use the codec features as they are. Further I am still not convinced to use in such a case an additional mechanism in the FU header.

Anyway, it is hard for me to imagine an advanced codec product, which does not has control over the codec to use tools such as tiles and/or dependent slice segments, where the latter tool is much easier to use and to apply to a bitstream.

--
Thomas Schierl
Fraunhofer HHI