Subject: Re: [payload] [AVTCORE] Clarifying the H.264 RTP payload format specification's text on when to set the RTP "M" bit

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 8 Aug 2013 16:40:15 +0000

I share Mo's opinion here that there is no need to add additional text to the H.264 or the H.265 draft in this aspect, rather the semantics of the M bit is sufficient.

First of all, implementers are expected to have basic knowledge of the video coding specification itself. If we try to explain all the coding-level details, there can be a lot.

Secondly, as I mentioned in the avtcore session in Berlin, the M bit for H.264 payload format is not reliable because RFC 6184 supports multi-time aggregation packets (MTAPs). This is not an issue any more in the H.265 payload as MTAPs are not supported anymore, thus the M bit can be a reliable indication. Of course, bad packetization/sender implementations can make that unreliable - but bad packetization/sender implementations can make anything unreliable - thus they don't count.

Lastly, the suggested text is too vague and even incorrect. For example, in the following copied part: what is "the current NAL unit"? Also, it is possible for non-VCL NAL unit to be between two VCL NAL units in an AU, the following language would suggest that the first of the two VCL NAL unit ends the AU - this is definitely incorrect.
    The current NAL unit ends an access unit if it is a VCL NAL unit, and if either:
            - the next NAL unit is not a VCL NAL unit, or
            - the next NAL unit is a VCL NAL unit, and the high-order bit of the first byte after its NALU header is 1.

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Sunday, August 04, 2013 2:52 PM
To: [email-masked]
Subject: Re: [AVTCORE] Clarifying the H.264 RTP payload format specification's text on when to set the RTP "M" bit

If you are willing to detect the last NALU of a frame by waiting for the first VCL NALU of the next frame, that is very simple in both H.264 and H.265. The first bit after the VCL NALU header is set if it's the first VCL NALU of a frame. This works in H.265 because the slice header explicitly has such a flag.

Thanks.  This is exactly the kind of information that would be useful for implementers.

Note that RTP receivers are technically forbidden to rely on the marker bit. But many implementations detect whether it is reliable, then use it as an optimization to minimize latency if reliable.

Yes, it's unfortunate that there may be implementations out there that do not set the RTP 'M' bit correctly.  But to reduce the likelihood of this happening, we should try to make our RTP payload format specifications as clear as is reasonable about when to set the 'M' bit.  (It is common - in non-interactive environments - for RTP transmitting applications to obtain their encoded data as a byte stream.)

 I don't think the draft needs any more text beyond what it currently says.

On the contrary - I think you've explained that some additional text would be both useful and reasonable.  (Once again, because we're not MPEG, we should not feel the need to fully-specify only the behavior that's required by RTP *receivers*.)

So I suggest adding the following paragraph to the H.265 RTP payload format specification, just after the existing 'M bit' text:

----------
Unfortunately the contents of a NAL unit, alone, does not tell a RTP sender implementation whether or not the NAL unit ends an access unit.  Instead, the implementation can obtain this information separately, from the encoder.  If, however, this information is not directly available from the encoder (e.g., because the implementation is sending data that consists solely of a sequence of pre-encoded NAL units), then it must instead inspect the next NAL unit, to determine whether or not the current NAL unit ends an access unit.  The following rule can be used:
    The current NAL unit ends an access unit if it is a VCL NAL unit, and if either:
          - the next NAL unit is not a VCL NAL unit, or
          - the next NAL unit is a VCL NAL unit, and the high-order bit of the first byte after its NALU header is 1.
----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: Ross Finlayson <[email-masked]>
Date: Thu, 8 Aug 2013 20:15:19 -0700

(I changed the "Subject:" line to make it clear that we're talking about the (future) H.265 RTP payload format specification here; not the (existing) H.264 RTP payload format specification.)

> First of all, implementers are expected to have basic knowledge of the video coding specification itself.

Yes.  However, in this case we seem to disagree on what constitutes 'basic knowledge'.  Determining whether or not a NAL unit ends an 'access unit' (in the situation - common for many implementations - where this is not available directly from the encoder) is far from 'basic knowledge' of H.265.  Unless you can point to a specific section in the H.265 specification that describes this (in which case we could add a reference to that section), it seems prudent to add a little text to our RTP payload format specification that clarifies this.

> Of course, bad packetization/sender implementations [of the RTP 'M' bit] can make that unreliable – but bad packetization/sender implementations can make anything unreliable – thus they don’t count.

I'm not sure what you're saying here.  I hope you're not saying that we shouldn't really care about whether or not implementations set the 'M' bit correctly, because in that case I definitely disagree; that's the whole point of this email thread.

>  Lastly, the suggested text is too vague and even incorrect.

Like Mohamed, you're inadvertently making my point for me :-)

> For example, in the following copied part: what is “the current NAL unit”?

By this I mean the NAL unit that's being packed in the RTP packet for which we are considering whether or not to set the "M' bit (or the final such NAL unit, if there's more than one).  

> Also, it is possible for non-VCL NAL unit to be between two VCL NAL units in an AU, the [suggested] language would suggest that the first of the two VCL NAL unit ends the AU – this is definitely incorrect.

Yes, good point - thanks.

Here, then, is new wording for the suggested paragraph (to be added to the H.265 RTP payload format specification, just after the existing 'M bit' text):

----------
Unfortunately the contents of a NAL unit, alone, does not tell a RTP sender implementation whether or not the NAL unit ends an access unit.  Instead, the implementation can obtain this information separately, from the encoder.  If, however, this information is not available directly from the encoder (e.g., because the implementation is sending data that consists solely of a sequence of pre-encoded NAL units), then it must instead inspect subsequent NAL units, to determine whether or not the NAL unit ends an access unit.  The following rule can be used:
    A NAL unit ends an access unit if it is a VCL NAL unit, and the next-occurring VCL NAL unit has the high-order bit of the first byte after its NALU header equal to 1.
----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 9 Aug 2013 13:01:02 +0000

Though I don't think it is not really necessary, I think we can either simply add a note after the semantics of the M bit saying that the first slice header bit indicates that the slice is the first slice of a picture, or mention this as part of the introduction to HEVC.

The addition is not really necessary because the first slice header bit is more expected to be used by video decoders for detection of the start of a new picture without relying on timing (because they might not be available to the decoder etc), while for the receivers at RTP level, they can always use the M bit and/or the RTP timestamp.

The semantics of the M bit is clearly specified, thus sender implementations should follow the semantics. Those implementations that do not follow what is specified, e.g. the M bit semantics, when encapsulating NAL units into packets, are bad implementations. It seems to me that the reasoning for your suggestion is that you expect in the future a lot of H.265 RTP payload format implementations would not follow the M bit. Nobody knows the future, but I hope this would not happen, as it is so easy to follow the semantics, and I don't see a reason why implementers would not follow what is being specified. As I explained twice already, not as in RFC 6184, in the context of the H.265 payload draft, there is no reliability issue with the M bit to be used for detection the end of an AU (after de-jittering of course - same for all approaches.

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Thursday, August 08, 2013 8:15 PM
To: [email-masked]; [email-masked]
Subject: Re: [AVTCORE] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Of course, bad packetization/sender implementations [of the RTP 'M' bit] can make that unreliable - but bad packetization/sender implementations can make anything unreliable - thus they don't count.

For example, in the following copied part: what is "the current NAL unit"?

Also, it is possible for non-VCL NAL unit to be between two VCL NAL units in an AU, the [suggested] language would suggest that the first of the two VCL NAL unit ends the AU - this is definitely incorrect.

----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 9 Aug 2013 15:47:44 +0000

Typo: "Though I don't think it is not really necessary" -> "Though I don't think it is really necessary" or "Though I think it is not really necessary".

From: [email-masked] [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: Friday, August 09, 2013 6:01 AM
To: Ross Finlayson; [email-masked]; [email-masked]
Subject: Re: [AVTCORE] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

BR, YK

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Thursday, August 08, 2013 8:15 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [AVTCORE] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: Ross Finlayson <[email-masked]>
Date: Fri, 9 Aug 2013 13:11:01 -0700

> Though I don’t think it is [] really necessary, I think we can either simply add a note after the semantics of the M bit saying that the first slice header bit indicates that the slice is the first slice of a picture, or mention this as part of the introduction to HEVC.

Because the new text is intended to help implementers figure out when to set the 'M' bit, it should be added just after the existing 'M' bit text.  (But why not just use the paragraph that I suggested?)

> The addition is not really necessary because the first slice header bit is more expected to be used by video decoders for detection of the start of a new picture without relying on timing (because…

Yes, but that's irrelevant here, because we're talking about how to clarify the text for implementers of RTP *senders*, not RTP receivers.  (If the current proposed RTP payload format specification were to be used only by the implementers of H.265/RTP receivers, then I would have no problem with it.  But that's not the case.) 

> it is so easy to follow the semantics, and I don’t see a reason why implementers would not follow what is being specified.

The reason is (as I've explained before) that for many RTP sender implementers who aren't very familiar with the H.265 codec specification, it will not be immediately obvious whether a NAL unit ends an access unit.  That's why I'm proposing adding a (very small) bit of extra text to clarify this.

I think the core of the disagreement here is that there are two different views on what a RTP payload format specification should be:

One view is that a RTP payload format specification should be the shortest, most concise possible document that describes how to send/receive RTP packets for a particular codec, for someone how is already very familiar with the codec specification.  From the perspective of a codec designer, this point of view makes perfect sense: The codec specification is the 'core document'; the RTP payload format specification is logically just an 'appendix'.  Therefore, from this point of view, there seems little sense in adding extra, redundant text to the RTP payload format document, because it's information that is already available in the codec specification.

However, there are two problems with this point of view.  The first problem is that it makes the specification 'fragile'.  Someone who happens to misread the codec specification is more likely to implement the RTP payload format incorrectly.

The second problem with this point of view is a practical one.  Many (if not most) people who implement RTP payload formats are not codec experts.  They just want to know how to transmit data that they've received from an encoder, or (on the receiving end) want to know how to feed data that they've received from a RTP stream into a decoder.  It's not reasonable to expect them to have intimate knowledge of the codec specification (especially since they may want to implement the RTP payload formats for several codecs - not just one).  Nor will they always have time to read through the codec specification in detail to find information that could just as easily have been made available in the RTP payload format specification.

An alternative view of what a RTP payload format should be (and this is the view that is, I think, is shared by most members of the AVT(CORE) working group) is that a RTP payload format specification should have sufficient detail to allow someone to implement the payload format (sending and/or receiving), even if they have only a limited knowledge of the codec specification.  Even if this means adding text to the RTP payload format specification that might - from the point of view of the codec designer - seem somewhat redundant.

I hope this helps you understand why adding just a small bit of redundant text (such as the paragraph I suggested) to the RTP payload format specification (after the existing 'M' bit text) will help implementers, and makes it more likely that they will implement the RTP payload format correctly.

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 9 Aug 2013 22:49:34 +0000

Now I understand your intention better - did not see that the note was intended for sender implementers.

For this purpose, indeed (as Mo pointed out earlier?) that first of all, the sender needs make sure the RTP timestamp for each RTP packet is correct. As long as RTP timestamp is correct, it is also easy to know when the M bit should be set. Or are you saying that there is no problem with RTP timestamp setting but there is a problem for the M bit setting? If yes, please clarify why.

I have no problem to putting some information in the line as you suggested, if the suggested text is good. However, the suggested wording below still has a few problems (or shortcomings).

----------

The problems (or shortcomings) are:

-          I don't think it is good to say that the fact that a NAL unit itself does not indicate whether it is the last one of an AU is unfortunate.

-          Saying that the information can be obtained from the encoder is vague to me. I guess you meant the bitstream instead, as the encoder may be absent e.g. when dealing pre-encoded video.

-          Saying that inspecting subsequent NAL units is a must is not really correct, as I believe timing information can be used too - as mentioned above, you got to make sure the RTP timestamp value is correctly set anyway.

-          Last but not least, an AU may also end with a non-VCL NAL unit, e.g. a filler data NAL unit, an end of sequence NAL unit, an end of bitstream NAL unit, a suffix SEI NAL unit, etc.

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Friday, August 09, 2013 1:11 PM
To: [email-masked]; [email-masked]
Subject: Re: [payload] [AVTCORE] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Though I don't think it is [] really necessary, I think we can either simply add a note after the semantics of the M bit saying that the first slice header bit indicates that the slice is the first slice of a picture, or mention this as part of the introduction to HEVC.

it is so easy to follow the semantics, and I don't see a reason why implementers would not follow what is being specified.

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: Ross Finlayson <[email-masked]>
Date: Fri, 9 Aug 2013 17:47:50 -0700

> Now I understand your intention better – did not see that the note was intended for sender implementers. 

OK, thanks.

>  For this purpose, indeed (as Mo pointed out earlier?) that first of all, the sender needs make sure the RTP timestamp for each RTP packet is correct. As long as RTP timestamp is correct, it is als…

Figuring out the RTP timestamp isn't (nearly as much of) a problem, because this can be done by knowing the stream's frame rate.  But in this email thread, I'm focusing only on the RTP 'M' bit, because for it (unlike the timestamp) there's less of an incentive for a sender implementation to 'get it right'.  I'm worried that if the RTP payload format specification doesn't make it crystal clear when a NAL unit ends an 'access unit', then an implementer might be tempted to just not bother setting the bit at all.  I don't want to see that happen.  Hence this email thread.

> -          I don’t think it is good to say that the fact that a NAL unit itself does not indicate whether it is the last one of an AU is unfortunate.

Why not?  It *is* "unfortunate".  I'm not intending to 'insult' H.265 here :-)

> -          Saying that the information can be obtained from the encoder is vague to me. I guess you meant the bitstream instead, as the encoder may be absent e.g. when dealing pre-encoded video.

I think you may have been confused by my use of the word "can" in the 2nd sentence of my proposed paragraph.  Here is a rewording of the 2nd (& 3rd) sentences; I hope this will make things clearer:

	"Instead, the implementation may be able to obtain this information separately, from the encoder.  If, however, the implementation cannot obtain this information directly from the encoder (e.g., because the implementation is sending data that consists solely of a sequence of pre-encoded NAL units), then it must instead inspect subsequent NAL units, to determine whether or not the NAL unit ends an access unit."

> -          Last but not least, an AU may also end with a non-VCL NAL unit, e.g. a filler data NAL unit, an end of sequence NAL unit, an end of bitstream NAL unit, a suffix SEI NAL unit, etc.

Arggh!  Now, I hope, you see the problem :-)  Can you then help me figure out a more accurate wording for the rule?  In particular:
	1/ Suppose we have a VCL NAL unit, followed immediately by another VCL NAL unit with the first slice header bit set.  In this case, there's no doubt: The first VCL NAL unit definitely ends an 'access unit' - correct?
	2/ Suppose we have a VCL NAL unit, followed by a sequence of one or more non-VCL NAL units, followed immediately by a VCL NAL unit with the first slice header bit set.  In this case, which (if any) of the non-VCL NAL units ends an 'access unit'?  Can we figure out a rule (ideally, an easy rule, based solely on the "nal_unit_type"s) for this?

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Sat, 10 Aug 2013 03:15:02 +0000

Hi Ross,

If we focus on the RTP timestamp rather than the marker bit, I think we will solve your problem. The timestamp MUST be set reliably since it is critical for many RTP functions. The marker bit SHOULD be set reliably to allow receivers to detect end of frame without waiting for the next frame/timestamp, but this is not a MUST because it is just a latency optimization rather than a critical function.

If you are unable to set the RTP timestamp reliably, that is a much bigger issue than the marker bit. That would be an entirely different discussion, so I assume that is not the case.

If you are able to set the RTP timestamp reliably, then you must already have some sort of frame boundary identification. In that case, you can set the marker bit based on the timestamp changing, assuming you are willing to wait for the next frame/timestamp before sending the last packet of a frame.

But you should realize that, in doing this, you have defeated the marker bit optimization. You have just shifted the latency from the receiver to the sender, but the latency remains the same. In both cases you incur the extra latency between the last packet of a frame and the first packet of the next frame.

If the sender incurs this latency to make the marker bit reliable, the receiver will hopefully detect that it is reliable and not incur further latency. However, a simple receiver may incur further latency if it only looks for timestamp changes and has no logic to detect marker reliability to optimize out this latency. Also, if the receiver jitter buffer is currently longer than the extra latency added by the sender, then the sender added the extra latency unnecessarily.

If the sender never sets the marker bit to avoid adding latency, the receiver will incur the latency once it sees the marker bit is unreliable (or unconditionally for simple receivers that only look for timestamp changes).

So there is not much benefit in making the marker bit reliable in this case, and there is even the risk that it will add unnecessary latency.

Regards,
Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Friday, August 09, 2013 8:48 PM
To: [email-masked]; [email-masked]
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

OK, thanks.

----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: Ross Finlayson <[email-masked]>
Date: Fri, 9 Aug 2013 21:49:45 -0700

> If you are able to set the RTP timestamp reliably, then you must already have some sort of frame boundary identification.

Yes, we have a reliable way of determining when *VCL* NAL units cross a frame boundary: Look at the first slice header bit.  The one remaining problem (that I identified at the end of my last message) is: How to classify any *non-VCL* NAL units that appear between one VCL NAL unit and another VCL unit that has the first slice header bit.  Which of these belong to the last frame, and which belong to the next frame.

But perhaps this doesn't really matter?  Perhaps it's OK (from the point of view of receivers that check and try to optimize for the 'M' bit) if we always set the 'M' bit for the last VCL NAL unit of a frame, even if some subsequent non-VCL NAL units (appearing before the first VCL NAL unit of the next frame) also belong logically to the current frame??

> But you should realize that, in doing this, you have defeated the marker bit optimization. You have just shifted the latency from the receiver to the sender

No, not necessarily.  If the encoded video comes from a file (i.e., pre-recorded), then the latency involved in requiring the sender to examine the 'next' frame will usually be negligible.  But in any case, the situation we're talking about here (senders working from a sequence of pre-encoded NAL units) is one that usually occurs only in non-interactive environments, for which latency isn't crucial.

> So there is not much benefit in making the marker bit reliable in this case, and there is even the risk that it will add unnecessary latency.

That's certainly another possibility, I suppose: Recommend that if a sending implementation cannot determine whether or not a NAL unit ends an 'access unit' (frame) without having to scan subsequent NAL units, then it should not set the 'M' bit at all.  I don't like this, though (especially because it adds receiver latency without necessarily reducing sender latency (note above)).  But it's a possibility to consider...

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Sat, 10 Aug 2013 13:54:45 +0000

Hi Ross,

You just need one more little step to get everything right here, instead of thinking of compromising the implementation :)

That is, read subclause 7.4.2.4.4 of the HEVC spec. The most relevant paragraphs are copied below for convenience (the first paragraph itself contains the answer to your question, but the second paragraph may also provide helpful information):

The first of any of the following NAL units after the last VCL NAL unit of a coded picture specifies the start of a new access unit:
-     access unit delimiter NAL unit (when present),
-     VPS NAL unit (when present),
-     SPS NAL unit (when present),
-     PPS NAL unit (when present),
-     Prefix SEI NAL unit (when present),
-     NAL units with nal_unit_type in the range of RSV_NVCL41..RSV_NVCL44 (when present),
-     NAL units with nal_unit_type in the range of UNSPEC48..UNSPEC55 (when present),
-     first VCL NAL unit of a coded picture (always present).

The order of the coded pictures and non-VCL NAL units within an access unit shall obey the following constraints:
-     When an access unit delimiter NAL unit is present, it shall be the first NAL unit. There shall be at most one access unit delimiter NAL unit in any access unit.
-     When any prefix SEI NAL units are present, they shall not follow the last VCL NAL unit of the access unit.
-     NAL units having nal_unit_type equal to FD_NUT or SUFFIX_SEI_NUT, or in the range of RSV_NVCL45..RSV_NVCL47 or UNSPEC56..UNSPEC63 shall not precede the first VCL NAL unit of the coded picture.
-     When an end of sequence NAL unit is present, it shall be the last NAL unit in the access unit other than an end of bitstream NAL unit (when present).
-     When an end of bitstream NAL unit is present, it shall be the last NAL unit in the access unit.

NOTE - VPS NAL units, SPS NAL units, PPS NAL units, prefix SEI NAL units, or NAL units with nal_unit_type in the range of RSV_NVCL41..RSV_NVCL44 or UNSPEC48..UNSPEC55, may be present in an access unit, but cannot follow the last VCL NAL unit of the coded picture within the access unit, as this condition would specify the start of a new access unit.

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Friday, August 09, 2013 9:50 PM
To: [email-masked]; [email-masked]
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Sat, 10 Aug 2013 14:16:46 +0000

Regardless of the following, I still think that the timestamp approach should be the way to go. A good system should have a clear mapping of each video data unit (NAL unit in the context of H.265 and H.264) to an output timestamp, be it real-time encoding or pre-encoded content. For real-time encoding, the encoder knows what data units are generated for each input raw picture and thus it clearly knows the timestamp for each data unit. A pre-encoded content should have been stored in some kind of video storage format that has timestamp associated with each data unit, e.g. in a file container according to ISO base media file format, MPEG-2 systems, or whatever other file storage format. Trying to generate timestamps from a raw coded video bitstream without timing information for each coded picture based on a guess of frame rate is really just a hack to me, as frame rate can be adaptive, particularly at the beginning of a session in low-delay conversational applications or at locations of scene change or of need of full intra refresh or random accessibility, wherein your rate control algorithm can easily require skipping the encoding of some frames.

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: Saturday, August 10, 2013 6:55 AM
To: Ross Finlayson; [email-masked]; [email-masked]
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Hi Ross,

BR, YK

---

From: Ross Finlayson <[email-masked]>
Date: Mon, 12 Aug 2013 12:25:23 -0700

> You just need one more little step to get everything right here, instead of thinking of compromising the implementation J
>  
> That is, read subclause 7.4.2.4.4 of the HEVC spec. The most relevant paragraphs are copied below for convenience (the first paragraph itself contains the answer to your question, but the second paragraph may also provide helpful information)

Thanks for the info; that was useful.

Here, then is the updated text of a paragraph - to be added after the existing 'M bit' text - that clarifies when/how to set it:
----------
Unfortunately the contents of a NAL unit, alone, do not tell a RTP sender implementation whether or not the NAL unit ends an access unit.  Instead, the implementation may be able to obtain this information separately, from the encoder.  If, however, the implementation cannot obtain this information directly from the encoder (e.g., because the implementation is sending data that consists solely of a sequence of pre-encoded NAL units), then it must instead inspect subsequent NAL units, to determine whether or not the NAL unit ends an access unit.  If the NAL units have already been timestamped, then these timestamps can be used to determine the access unit boundaries.  Otherwise, the access unit boundaries (and times) must be determined by inspecting the NAL units' contents - specifically, the "nal_unit_type" and (for VCL NAL units) the "first_slice_segment_in_pic_flag".  The following rule can be used:
	 A NAL unit (X) ends an access unit if the next-occurring VCL NAL unit (Y) has the high-order bit of the first byte after its NAL unit header equal to 1, and all NAL units, if any, between (X) and (Y) have a "nal_unit_type" in one of the following ranges: [32,35], 39, [41,44], or [48,55].
----------

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 27 Aug 2013 04:50:49 +0000

Hi Ross,

Sorry for a slow response. I started to edit this draft today, and have included the following note after the semantics of the M bit:

Informative note: The content of a NAL unit does not tell whether or not the NAL unit is the last NAL unit, in decoding order, of an access unit.  An RTP sender implementation may obtain this information from the video encoder.  If, however, the implementation cannot obtain this information directly from the encoder, e.g., when the stream was pre-encoded, and also there is no timestamp allocated for each NAL unit, then the sender implementation can inspect subsequent NAL units in decoding order to determine whether or not the NAL unit is the last NAL unit of an access unit as follows.  A NAL unit naluX is the last NAL unit of an access unit if the next VCL NAL unit naluY in decoding order has the high-order bit of the first byte after its NAL unit header equal to 1, and all NAL units between naluX and naluY, when present, have nal_unit_type in the range of 32 to 35, inclusive, equal to 39, in the range of 41 to 44, inclusive, or in the range of 48 to 55, inclusive.

Would you be fine with this wording?

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross Finlayson
Sent: Monday, August 12, 2013 12:25 PM
To: [email-masked]; [email-masked]
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: Ross Finlayson <[email-masked]>
Date: Mon, 26 Aug 2013 22:38:53 -0700

> Would you be fine with this wording?

Yes, this looks good to me.

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 27 Aug 2013 22:58:04 +0000

I suggest replacing "The content of a NAL unit" with "The NAL unit header". The content does reveal end of access unit boundaries if you're willing to parse/decode it all.

Mo

On Aug 27, 2013, at 12:51 AM, "Wang, Ye-Kui" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Ross,

BR, YK

You just need one more little step to get everything right here, instead of thinking of compromising the implementation ☺

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

_______________________________________________
Audio/Video Transport Core Maintenance
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/avt

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 28 Aug 2013 01:36:20 +0000

Hi Mo,

"The content of a NAL unit" can tell whether the NAL unit is the last slice of a picture, but the last NAL unit of an AU can also be a non-VCL NAL unit. Considering this, "the content of a NAL unit" does not tell whether a NAL unit is the last NAL unit of an AU.

BR, YK

From: Mo Zanaty (mzanaty) [mailto:[email-masked]]
Sent: Tuesday, August 27, 2013 3:58 PM
To: Wang, Ye-Kui
Cc: Ross Finlayson; [email-masked]; [email-masked]
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Mo

BR, YK

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 28 Aug 2013 04:21:36 +0000

Hi YK,

Good point on trailing non-VCL NAL units.

That brings up another point on non-VCL NAL units. I assume the non-VCL NAL units defined in this draft (AP/FU) can’t appear in the bitstream processed by the RTP sender using this informative note. That is, the RTP sender must be processing an elementary stream that only contains NAL units specified in the H.265 standard itself [0-47], and can’t be processing a bitstream that came from another RTP sender which may have inserted unspecified NAL units [48-63] like AP/FU, since that could yield improper results for this note.

Or do we actually need to split AP and FU each into two non-VCL units, one in the leading non-VCL range [48-55] and another in the trailing non-VCL range [56-63]? I would like to avoid that complexity, but that seems to be what the rules in H.265 7.4.2.4.4 require when using the unspecified range [48-63]. Do you recall the motivation for rules on subranges of the unspecified range?

Also, it may be useful to indicate in the informative note how to differentiate VCL NAL units [0-31] from non-VCL NAL units [32-63], i.e. the MSB of the 6-bit NAL unit type is 1 in non-VCL NAL units.

Regards,
Mo

From: Wang, Ye-Kui [mailto:[email-masked]]
Sent: Tuesday, August 27, 2013 9:36 PM
To: Mo Zanaty (mzanaty)
Cc: Ross Finlayson; [email-masked]; [email-masked]
Subject: RE: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Hi Mo,

BR, YK

Mo

BR, YK

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 28 Aug 2013 17:55:11 +0000

Hi Mo,

Good point. The note is really about an RTP sender implementation takes an HEVC bitstream as input and generates RTP packets as the output, while usually NAL units with type in the range of [48-63] would not be present in an HEVC bitstream. Hypothetically, even some sorts of NAL units with type in the range of [48-63] are present, the note is still correct in terms of telling which NAL unit is the last NAL unit of an AU. Note that from the video coding spec point of view, the range of [48-63] is not just left unspecified for the RTP payload formats, but also for file formats and other systems components as well.

In JCT-VC, the splitting of the range [48-63] into two subcategories, prefix and suffix, was a result from a discussion on the PACSI NAL unit in RFC 6190, which is effectively a prefix, but in H.264/AVC all the “unspecified” NAL unit types are specified as suffix. The issue was pretty hypothetical IMO, as anyway an RTP depacketizer (or a file parser) would not feed stuff like APs or FUs directly into a decoder. One related action we should probably do is to explicitly disallow an RTP depacketizer to directly output APs or FUs without depacketizing them even though that looks obvious.

Currently in this draft, APs and FUs use types 48 and 49, two of the prefix “unspecified” NAL unit types. This is fine, as in an RTP stream an AP or FU may start an “AU”.

Regarding your last point, I agree that indeed it would be useful to explicitly spell out which NAL units are VCL NAL units (and which ones are non-VCL NAL unit) in this draft. Instead of putting this information into the note, I think it is probably even better to mention it in Section 1.1.4 (NAL Unit Header) as part of the introduction to the nal_unit_type field.

BR, YK

From: Mo Zanaty (mzanaty) [mailto:[email-masked]]
Sent: Tuesday, August 27, 2013 9:22 PM
To: Wang, Ye-Kui
Cc: Ross Finlayson; [email-masked]; [email-masked]
Subject: RE: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Hi YK,

Regards,
Mo

From: Wang, Ye-Kui [mailto:[email-masked]]
Sent: Tuesday, August 27, 2013 9:36 PM
To: Mo Zanaty (mzanaty)
Cc: Ross Finlayson; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Hi Mo,

BR, YK

From: Mo Zanaty (mzanaty) [mailto:[email-masked]]
Sent: Tuesday, August 27, 2013 3:58 PM
To: Wang, Ye-Kui
Cc: Ross Finlayson; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [AVTCORE] [payload] Clarifying the H.265 RTP payload format specification's text on when to set the RTP "M" bit

Mo

BR, YK

Ross Finlayson
Live Networks, Inc.
http://www.live555.com/