Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

From: Magnus Westerlund <[email-masked]>
Date: Fri, 2 May 2014 10:57:36 +0200

On 2014-04-30 16:40, Yago Sanchez wrote:
> Hi Magnus,
> 
> I think there is a misunderstanding with the intention of using the
> profile-compatibility-indicator. It is not about indicating support for
> additional profiles. The profile-compatibility-indicator sets additional
> constraints to the profile indicated in the profile-id. That is why
> Ye-Kui was pointing that both the profile-id and the
> profile-compatibility-indicator are used to express the same as  the
> profile_idc in RFC 6184. This means that if the offerer indicates
> profile-id equal to A and includes profile B in the
> profile-compatibility-indicator, the offerer would indicate capability
> of encoding/decoding a stream that only contains the tools that are in
> the intersection of both profiles. The offerer promise would not even be
> that it is able to decode the full A profile, unless indicated in
> another PT.

No, I understand that this in HEVC spec indicates that although a
particular coded video sequence where the parameters in included is
compliant not only to the profile it is declared for in the profile
indicator parameter, it is also compliant to these marked in the
profile-compatibility-indicator.

However, what applies to a specific encoder instances and more precisely
a coded video sequence (CVS) has a different meaning if one applies
these constraints to a RTP payload type. An RTP payload type has
applicability across all RTP packet streams that claims to use this
payload type.

In addition in an SDP O/A context with the rules currently in the draft,
i.e. they apply both to the offerer and the answerer if they maintain
the payload type, that means that from a P2P session context, both
endpoints if accepting the PT, must be able to encoded and output a CVS
which fulfils the requirements of the profile-compatibility-indicator.
Which means that you MUST have an encoder that both are aware of all the
profiles indicated in the included instance of
profile-compatibility-indicator, i.e. you can't ignore a new profile
that you don't know anything about, if the offerer has marked this
unknown profile as being in the compability set. You also as an answerer
must be able to determine that you can constrain your encoder
implementation to the same set of coding tools that the answerer
implementation have.

> 
> My understanding was that this would be useful for those cases where two
> endpoints support different profiles but are aware of the intersection
> with other profiles, as in the case expressed above. Thus, there could
> express more PTs with the profile they fully support and with subsets of
> it result of intersections with other profiles.

I think you will have to be more explicit how this is being done within
the rules of the RTP payload format for HEVC. I don't see how you an
answerer within SDP O/A rules can accept a payload type that has a
profile=X value where it doesn't support X, but do support Y that is
marked within profile-compatibility-indicator. The reason is that if the
answerer indicates support in a SendRecv m= section then it must be able
to decode profile X, it also must be able to send a stream marked as
being X, but fulfilling also Y. That is not the same as being able to
send a stream according to profile Y.

The issue I am trying to make you aware of is that the combination of
sender and receiver constraints profile-compatibility-indicator
introduces does not appear to provide any benefit, only limitations that
makes things less compatible.

> 
> Therefore, I still think that including profile-compatibility-indicator
> may be useful. Would this be reasonable or is there anything I am
> missing from the discussion below?

I think you are missing something about what it implies to declare a PT
in a sendrecv context with this parameter. It might be me missing
something, but I would like you to try to explain in detail how what you
think is possible above actually works.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Yago Sanchez <[email-masked]>
Date: Sun, 4 May 2014 22:32:31 +0200

Hi Magnus,

I still think that profile-compatibility-indicator may be useful. Let me list the different options I see and come finally to a conclusion. All options are related to offerer supporting profile A and answerer profile B.

1) Offerer includes profile A and no profile-compatibility-indicator. Answerer cannot accept profile A so they cannot achieve interoperability.

2) Offerer includes profile A and sets the flag related to profile B in the profile-compatibility-indicator (Offerer has to be aware of the intersection of both profiles). Answerer ist not aware of profile A and does not know which are the tools that lay in the intersection, so again interoperability cannot be achieved, since the answerer cannot constraint the sent stream so that the offerer can decode it.
Still, the use of profile-compatibility-indicator does not harm session negotiation. The reason for not achieving interoperability is another one.

3) Offerer includes profile A and sets the flag related to profile B in the profile-compatibility-indicator (Offerer has to be aware of the intersection of both profiles). Answerer ist also aware of profile A and the intersection with B and can accept the offer and answer with a stream to the offerer that can be decoded.
In this case profile-compatibility-indicator helps for achieving interoperability.

4) The last option might be one that could show a case where profile-compatibility-indicator could be harmful. Lets assume profile A is a superset of profile B. The answerer could offer profile A and set the flag corresponding to profile B in profile-compatibility-indicator or even offer profile B and set the flag corresponding to profile A in profile-compatibility-indicator (both cases express the same). The answerer even though it could decode and encode such a stream would have to reject the payload type if it is not aware of what means to have a stream being compatible to profile A due to not knowing profile A. I think this is what you are concern about right?

In summary, I only see it could be harmful in case 4. And case 4 is only the case where a full profile is supported (in the example profile B) and all profiles that are supersets of it are marked in the profile-compatibility-indicator. All other cases of profiles not being supersets and being marked in profile-compatibility-indicator would be either example 2 or 3.

Therefore my suggestion would be to still keep profile-compatibility-indicator, since it might achieve interoperability as shown in example 3 and add some text that expresses that if a profile is fully supported and profile-compatibility-indicator does not constraint that profile further the profile-id should indicate such a profile and profile-compatibility-indicator should not be included. Obviously this only applies for O/A and not for declarative session description. 

Would this make sense for you Ye-Kui and Magnus?

Best regards,
Yago

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 5 May 2014 10:04:58 +0200

Hi Yago,

Please see inline for comments

On 2014-05-04 22:32, Yago Sanchez wrote:
> Hi Magnus,
> 
> I still think that profile-compatibility-indicator may be useful. Let me
> list the different options I see and come finally to a conclusion. All
> options are related to offerer supporting profile A and answerer profile B.
> 
> 1) Offerer includes profile A and no profile-compatibility-indicator.
> Answerer cannot accept profile A so they cannot achieve interoperability.
> 
> 2) Offerer includes profile A and sets the flag related to profile B in
> the profile-compatibility-indicator (Offerer has to be aware of the
> intersection of both profiles). Answerer ist not aware of profile A and
> does not know which are the tools that lay in the intersection, so again
> interoperability cannot be achieved, since the answerer cannot
> constraint the sent stream so that the offerer can decode it.
> Still, the use of profile-compatibility-indicator does not harm session
> negotiation. The reason for not achieving interoperability is another one.
> 
> 3) Offerer includes profile A and sets the flag related to profile B in
> the profile-compatibility-indicator (Offerer has to be aware of the
> intersection of both profiles). Answerer ist also aware of profile A and
> the intersection with B and can accept the offer and answer with a
> stream to the offerer that can be decoded.

Lets look at this depending on directional attribute.

Offer: a=recvonly:
In this case, there was no point of offering the
profile-compatibility-indicator, as that would only unnecessary restrict
the answerer in its sending of the stream.

Offer: a=sendonly
So in this case the offerer can include the
profile-compatibility-indicator and the answerer will know that the
offered stream is also compliant with the marked profiles. However, I
seriously question if the answerer can accept this RTP payload type.
Yes, it can receive this specific stream under these constraints.
However, the receiver if it accepts the RTP payload type, it promises to
accept any stream that matches the RTP payload type. Thus, if one sends
an additional stream under this RTP payload type what are the promise.
It can't receive any stream fulfilling the profile-id parameter, it MUST
also follow the constraints of the profile-compatibility-indicator. So
what happens if that second encoder does not know about this profile.
Then you end up in a incompatibility. Sure, with tight enforcement of
that any stream you are being sent must fulfil to have the same
profile-compatibility-indicator as negotiated, it can work.

Offer: a=sendrecv
Answerer: a=recvonly

Equal to above

Offer: a=sendrecv
Answerer: a=sendonly
The answerer will not accept to receive the stream from Offerer, thus
the answerer can accept the payload type, given that it can provided a
stream matching the RTP payload type. However, as in this scenario the
answerer don't support the offered profile-id, only a another supported
profile. Thus, it doesn't fulfil the payload type and must reject it.

Offer: a=sendrecv
Answerer: a=sendrecv

In this case the answerer must reject the RTP payload on the grounds
that it can't send the offerer a stream matching the RTP payload type,
see just above with answerer:sendonly.

Thus, I would say that it provides some benefit in the cases the offerer
will only send the stream to answerer, but in no other cases. Also, this
might give the offerer the wrong impression, and if one tries to modify
the session to bi-directional streams, then this fails per the other
options.

Lets instead look at the option to include one payload type per
compatible profile that the encoder can send, i.e. each profile marked
in profile-compatibility-indicator will get its own RTP payload type:

Offer: a=recvonly:
Given that the decoder has the capability for all profiles the encoder
could produce, the compatibility would be maximized. However, this is
mostly irrelvant in this comparision.

Offer: a=sendonly
If the offerer includes both the intended profile and one for the
compatible, then the answer rejects the intended one and accepts the PT
for the compatible one. Things work, given that the answerer accepts a
bitstream with profile-Id = intended and
profile-compatibility-indicator=compatible one.

Offer: a=sendrecv
Answerer: a=recvonly
Equal to above with the additional benefit that the offerer knows which
profile the answerer supports to receive and likely send.

Offer: a=sendrecv
Answerer: a=sendonly
To be able to offer this, the offerer must also be able to receive the
compatible profile(s). If it doesn't then it can't offer multiple
profiles and then we anyway have compatibility rejection.

Offer: a=sendrecv
Answerer: a=sendrecv
To be able to offer this, the offerer must also be able to receive the
compatible profile(s). If not we have failure as there is no compatible
profile both can produce. If it can we have interoperability, and
answerer indicates that it can only produce the other profile.

You might now better understand why I bring this up. It mostly fails or
appear to have none intentional restrictions.

So I will admit that profile-compatibility-indicator provides a benefit
if, and only if one offers sendonly streams, or ends up in sendonly from
the party offering the profile-compatibility-indicator. This appears to
indicate that having this as a sender constraint only would reduce the
cases where it restricts. However, in this case one do loose the
possibility to negotiate for a central mixer case, that you must produce
a stream that fulfils the subset of these profiles. I do note that it
appear necessary to make it clear that one MUST accept a video bitstream
as long as its profile-id or profile-compatibility-indicator indicates
the profile one has negotiated for the payload type. Is that clear in
the HEVC specification? Does it need to be repeated in the payload
format to prevent packetizers and others to make this error if they look
slightly into the bitstream?

However, if we are getting into this detail of negotiation, then I would
suggest that we create one profile-compatibility-indicator signalling
attribute per send and receive direction. So that one can have senders
constraints without the receive constraints.

> 
> 4) The last option might be one that could show a case where
> profile-compatibility-indicator could be harmful. Lets assume profile A
> is a superset of profile B. The answerer could offer profile A and set
> the flag corresponding to profile B in profile-compatibility-indicator
> or even offer profile B and set the flag corresponding to profile A in
> profile-compatibility-indicator (both cases express the same). The
> answerer even though it could decode and encode such a stream would have
> to reject the payload type if it is not aware of what means to have a
> stream being compatible to profile A due to not knowing profile A. I
> think this is what you are concern about right?

It is one of my concerns, and depends on interpretation of the
profile-compatibility-indicator. This is only relevant if one is forced
to use the profile-compatibility-indicator as send direction constraint
in an encoder. In a reception only case, one can look for just a profile
one supports. Which further speaks for splitting the send and receiver
constrints.

> 
> In summary, I only see it could be harmful in case 4. And case 4 is only
> the case where a full profile is supported (in the example profile B)
> and all profiles that are supersets of it are marked in the
> profile-compatibility-indicator. All other cases of profiles not being
> supersets and being marked in profile-compatibility-indicator would be
> either example 2 or 3.
> 
> Therefore my suggestion would be to still keep
> profile-compatibility-indicator, since it might achieve interoperability
> as shown in example 3 and add some text that expresses that if a profile
> is fully supported and profile-compatibility-indicator does not
> constraint that profile further the profile-id should indicate such a
> profile and profile-compatibility-indicator should not be included.
> Obviously this only applies for O/A and not for declarative session
> description. 

I hope my above extension of your cases allows you to better see my
concerns with the current specification of the
profile-compatibility-indicator. If one would divide them into sender
and receive direction some of the issues goes away.

Cheers

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 6 May 2014 00:38:37 +0000

Hi Magnus, Yago,

Thanks for having the discussion ongoing. See my comments inline below. 

For easy of discussion, I numbered the different cases. I am hoping that through these discussions, we all would have a more comprehensive, and on-the-same-page, understanding on this issue. Then it should be easy to conclude how to proceed.

BR, YK

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Monday, May 05, 2014 1:05 AM
To: Yago Sanchez
Cc: Wang, Ye-Kui; [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi Yago,

On 2014-05-04 22:32, Yago Sanchez wrote:
> Hi Magnus,
> 
> I still think that profile-compatibility-indicator may be useful. Let 
> me list the different options I see and come finally to a conclusion. 
> All options are related to offerer supporting profile A and answerer profile B.
> 
> 2) Offerer includes profile A and sets the flag related to profile B 
> in the profile-compatibility-indicator (Offerer has to be aware of the 
> intersection of both profiles). Answerer ist not aware of profile A 
> and does not know which are the tools that lay in the intersection, so 
> again interoperability cannot be achieved, since the answerer cannot 
> Still, the use of profile-compatibility-indicator does not harm 
> session negotiation. The reason for not achieving interoperability is another one.
> 
> 3) Offerer includes profile A and sets the flag related to profile B 
> in the profile-compatibility-indicator (Offerer has to be aware of the 
> intersection of both profiles). Answerer ist also aware of profile A 
> and the intersection with B and can accept the offer and answer with a 

Case 1: Offer: a=recvonly:
In this case, there was no point of offering the profile-compatibility-indicator, as that would only unnecessary restrict the answerer in its sending of the stream.

[YK]: There can be still be a point to offer profile-compatibility-indicator in this case. For example, it is possible that the decoder of the offerer only supports the common tools of profiles A and B, while it does not support either full profile A or full profile B. Think about similar situation for the AVC Constrained Baseline profile, which contains the common tools of the Baseline profile and the Main profile, and was defined after the definitions of the Baseline profile and the Main profile, and uses the same profile_idc as the Baseline profile. Similar situations could happen for HEVC. Note that this can also happen when the direction attribute has other values.

Case 2: Offer: a=sendonly
So in this case the offerer can include the profile-compatibility-indicator and the answerer will know that the offered stream is also compliant with the marked profiles. However, I seriously question if the answerer can accept this RTP payload type.
Yes, it can receive this specific stream under these constraints.
However, the receiver if it accepts the RTP payload type, it promises to accept any stream that matches the RTP payload type. Thus, if one sends an additional stream under this RTP payload type what are the promise.
It can't receive any stream fulfilling the profile-id parameter, it MUST also follow the constraints of the profile-compatibility-indicator. So what happens if that second encoder does not know about this profile.
Then you end up in a incompatibility. Sure, with tight enforcement of that any stream you are being sent must fulfil to have the same profile-compatibility-indicator as negotiated, it can work.

[YK]: I see your point. In this case, if the offerer both profile A and profile B (by profile-id together with profile-compatibility-indicator), while the answerer understands profile A but does not understand profile B, the answerer can actually still accept the offer for the video communication from the offerer-to-answerer direction. However, for the other direction (answerer-to-offerer direction), the answerer does not know which tools to disable when encoding the video, there is no way to ensure interoperable video media in this direction. So, this can only work for single-direction communication, or declarative mode.

Case 3: Offer: a=sendrecv
Answerer: a=recvonly

Equal to above

[YK]: Agreed, and my above comments apply.

Case 4: Offer: a=sendrecv
Answerer: a=sendonly
The answerer will not accept to receive the stream from Offerer, thus the answerer can accept the payload type, given that it can provided a stream matching the RTP payload type. However, as in this scenario the answerer don't support the offered profile-id, only a another supported profile. Thus, it doesn't fulfil the payload type and must reject it.

[YK]: I don't understand the first part above (thus the answerer can accept the payload type ...) - maybe there is a typo. But you second part seems to be aligned with my understanding, which is as follows (just to make the example clearer such that we are talking about the same thing). In this case, if the offerer indicates profile A and profile B (by profile-id together with profile-compatibility-indicator), while the answerer understands profile A but does not understand profile B, since the answerer does not know which tools to disable when encoding the video, there is no way to ensure interoperable video media in this direction, thus it has to reject the payload type.

Case 5: Offer: a=sendrecv
Answerer: a=sendrecv

In this case the answerer must reject the RTP payload on the grounds that it can't send the offerer a stream matching the RTP payload type, see just above with answerer:sendonly.

Thus, I would say that it provides some benefit in the cases the offerer will only send the stream to answerer, but in no other cases. Also, this might give the offerer the wrong impression, and if one tries to modify the session to bi-directional streams, then this fails per the other options.

[YK]: The conclusion here should be slightly changed per Case 1. One optimization we can do here is to relax the constraint (that both profile_id and profile-compatibility-indicator must remain unchanged in offer and answer) a bit to enable Cases 2 and 3 (for single-direction from offerer to answerer, or declarative usage). For other cases, the current constraint applies (or as I said earlier, profile_id and profile-compatibility-indicator in both offer and answer must indicate the same set of encoding tools supported).

Lets instead look at the option to include one payload type per compatible profile that the encoder can send, i.e. each profile marked in profile-compatibility-indicator will get its own RTP payload type:

Case 6: Offer: a=recvonly:
Given that the decoder has the capability for all profiles the encoder could produce, the compatibility would be maximized. However, this is mostly irrelvant in this comparision.

[YK]: See discussions above for Case 1. This case can actually be split into two cases. Case 6.1: if the decoder does fully support each of the multiple profiles, then using one payload type for each of these profiles is OK. Case 6.2: if the decoder only supports the common tools of a list of profiles but not fully for each of the profiles, then using one payload type for each of these profiles is NOT OK.

Case 7: Offer: a=sendonly
If the offerer includes both the intended profile and one for the compatible, then the answer rejects the intended one and accepts the PT for the compatible one. Things work, given that the answerer accepts a bitstream with profile-Id = intended and profile-compatibility-indicator=compatible one.

[YK]: My above comments for Case 6 apply.

Case 8: Offer: a=sendrecv
Answerer: a=recvonly
Equal to above with the additional benefit that the offerer knows which profile the answerer supports to receive and likely send.

[YK]: My above comments for Case 6 apply.

Case 9: Offer: a=sendrecv
Answerer: a=sendonly
To be able to offer this, the offerer must also be able to receive the compatible profile(s). If it doesn't then it can't offer multiple profiles and then we anyway have compatibility rejection.

[YK]: My above comments for Case 6 apply.

Case 10: Offer: a=sendrecv
Answerer: a=sendrecv
To be able to offer this, the offerer must also be able to receive the compatible profile(s). If not we have failure as there is no compatible profile both can produce. If it can we have interoperability, and answerer indicates that it can only produce the other profile.

[YK]: My above comments for Case 6 apply.

You might now better understand why I bring this up. It mostly fails or appear to have none intentional restrictions.

So I will admit that profile-compatibility-indicator provides a benefit if, and only if one offers sendonly streams, or ends up in sendonly from the party offering the profile-compatibility-indicator. This appears to indicate that having this as a sender constraint only would reduce the cases where it restricts. However, in this case one do loose the possibility to negotiate for a central mixer case, that you must produce a stream that fulfils the subset of these profiles. I do note that it appear necessary to make it clear that one MUST accept a video bitstream as long as its profile-id or profile-compatibility-indicator indicates the profile one has negotiated for the payload type. Is that clear in the HEVC specification? Does it need to be repeated in the payload format to prevent packetizers and others to make this error if they look slightly into the bitstream?

However, if we are getting into this detail of negotiation, then I would suggest that we create one profile-compatibility-indicator signalling attribute per send and receive direction. So that one can have senders constraints without the receive constraints.

[YK]: I hope with my above comments, particularly those under Case 1, some of the conclusions above would be adjusted. And also please check my suggestion in the comments under Case 5.

> profile-compatibility-indicator could be harmful. Lets assume profile 
> A is a superset of profile B. The answerer could offer profile A and 
> set the flag corresponding to profile B in 
> profile-compatibility-indicator or even offer profile B and set the 
> flag corresponding to profile A in profile-compatibility-indicator 
> (both cases express the same). The answerer even though it could 
> decode and encode such a stream would have to reject the payload type 
> if it is not aware of what means to have a stream being compatible to 
> profile A due to not knowing profile A. I think this is what you are concern about right?

It is one of my concerns, and depends on interpretation of the profile-compatibility-indicator. This is only relevant if one is forced to use the profile-compatibility-indicator as send direction constraint in an encoder. In a reception only case, one can look for just a profile one supports. Which further speaks for splitting the send and receiver constrints.

[YK]: This is the same as Cases 4 and 5 above.

> 
> In summary, I only see it could be harmful in case 4. And case 4 is 
> only the case where a full profile is supported (in the example 
> profile B) and all profiles that are supersets of it are marked in the 
> profile-compatibility-indicator, since it might achieve 
> interoperability as shown in example 3 and add some text that 
> expresses that if a profile is fully supported and 
> profile-compatibility-indicator does not constraint that profile 
> further the profile-id should indicate such a profile and profile-compatibility-indicator should not be included.

I hope my above extension of your cases allows you to better see my concerns with the current specification of the profile-compatibility-indicator. If one would divide them into sender and receive direction some of the issues goes away.

[YK]: I think I now do see your points in Case 2 and 3, and as I said above, we can do some optimization (see again my suggestion above under Case 2). However, considering my above comments, particularly those under Case 1, I hope we can have a more comprehensive, and on-the-same-page understanding on this issue.

---

From: Yago Sanchez <[email-masked]>
Date: Tue, 6 May 2014 10:05:40 +0200

Hi Magnus, 

Let me bring the discussion on recv-sub-layer-id back so that I understand your point here. (removed the rest of the discussion for clarity).
Following the example Ye-Kui provided with two operation points (sub-layers) 720p@30fps and 720p@15fps, lets assume the offerer includes a payload type with profile-id and level-id for 720p@30 fps.

What do you mean when you say “the recv-sub-layer-id is fine as a stream specific selection mechanism, but does not replace the capability declaration the regular parameters perform”?  Does it mean that if the answerer accepts the payload type with recv-sub-layer-id equal to 0 in the answer (without level downgrade), it is selecting 720p@15fps but actually is indicating to have capabilities that match the profile-id and level-id of 720p@30 fps?

Best regards,
Yago

On 25 Apr 2014, at 19:56, Wang, Ye-Kui <[email-masked]> wrote:
> 
>> [YK]: Let's use an example, which should help. Let's say the offer SDP 
>> includes a VPS that provides two operation points, 720p@15fps and 
>> 720p@30fps, corresponding to recv-sub-layer-id 0 and 1, respectively.
>> The profile, tier, level, and so on for each operation point are 
>> signalled by the VPS. If recv-sub-layer-id=0 is included in an SDP 
>> answer, then that means the answerer chooses to operate at 720p@15fps. 
>> In this case, the profile, tier, level, and so on for the chosen 
>> operation point does not need to be explicitly sent again in the SDP, 
>> as it is clear what they are (they are signalled in the VPS in the SDP 
>> offer). So there is no ambiguity here. Are you suggesting that in some 
>> use cases even there is no ambiguity the configuration information 
>> must always be explicitly signalled (i.e. values that are derivable)? 
>> If yes, could you please explain in what use cases and why. Note that 
>> we had the same mechanism specified for SVC in RFC 6190. Also, on PT 
>> use, we have allowed the use of a different PT value in an SDP answer 
>> than the matching PT value in the offer since at least RFC 3984 as 
>> long as the PT matching is clear.
> 
> Okay, I can now put the finger on what troubles me with this. I think this is violation or at least an omission of the principle of SDP Offer/Answer. The reason is that you substitute the answering explicitly declaring its receiving capabilities with a reference to a particular instantiation of a video sequence parameter set that falls within the receiver's capabilities. Thus, the offerer does not know the receiver's actually capabilities if it needs to reconfigure the encoder, or in case the RTP payload type is going to be used by multiple media streams and possibly different encoder instances.
> 
> In addition it creates an issue for any middlebox that is on the signalling path that it keeps track and state for what was in the offer to know what the answer means.
> 
> Thus, I think it is important that the answerer doesn't treat this situation differently, and instead declare its capabilities as normally in answer. The recv-sub-layer-id parameter is an optional parameter and it is also stream specific, while the profile, level, etc are generically applicable to the RTP payload type do applies on RTP session level.
> 
> To conclude, the recv-sub-layer-id is fine as a stream specific selection mechanism, but does not replace the capability declaration the regular parameters perform.
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 6 May 2014 11:28:26 +0200

On 2014-05-06 10:05, Yago Sanchez wrote:
> Hi Magnus,
> 
> Let me bring the discussion on recv-sub-layer-id back so that I
> understand your point here. (removed the rest of the discussion for
> clarity). Following the example Ye-Kui provided with two operation
> points (sub-layers) 720p@30fps and 720p@15fps, lets assume the
> offerer includes a payload type with profile-id and level-id for
> 720p@30 fps.
> 
> What do you mean when you say “the recv-sub-layer-id is fine as a
> stream specific selection mechanism, but does not replace the
> capability declaration the regular parameters perform”?  Does it mean
> that if the answerer accepts the payload type with recv-sub-layer-id
> equal to 0 in the answer (without level downgrade), it is selecting
> 720p@15fps but actually is indicating to have capabilities that match
> the profile-id and level-id of 720p@30 fps?
> 

Yes, that is the crux of the issue. The payload type states that you
have the 720p@30fps capability, as you haven't changed your profile or
level. While receive-sub-layer-id can point to that lower capability
720p@15fps. That is a contradiction.

In addition using a pointer to value in the offer, results that the
actual value is not directly available when parsing the answer.

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 6 May 2014 11:56:12 +0200

On 2014-05-06 02:38, Wang, Ye-Kui wrote:
> For easy of discussion, I numbered the different cases. I am hoping
> that through these discussions, we all would have a more
> comprehensive, and on-the-same-page, understanding on this issue.
> Then it should be easy to conclude how to proceed.
> 
> BR, YK
> 
> -----Original Message----- From: Magnus Westerlund
> [mailto:[email-masked]] Sent: Monday, May 05, 2014
> 1:05 AM To: Yago Sanchez Cc: Wang, Ye-Kui; [email-masked];
> [email-masked] Subject: Re: [payload]
> Comments on draft-ietf-payload-rtp-h265-02
> 
> Hi Yago,
> 
>> I still think that profile-compatibility-indicator may be useful.
>> Let me list the different options I see and come finally to a
>> conclusion. All options are related to offerer supporting profile A
>> and answerer profile B.
>> 
>> 1) Offerer includes profile A and no
>> profile-compatibility-indicator. Answerer cannot accept profile A
>> so they cannot achieve interoperability.
>> 
>> 2) Offerer includes profile A and sets the flag related to profile
>> B in the profile-compatibility-indicator (Offerer has to be aware
>> of the intersection of both profiles). Answerer ist not aware of
>> profile A and does not know which are the tools that lay in the
>> intersection, so again interoperability cannot be achieved, since
>> the answerer cannot constraint the sent stream so that the offerer
>> can decode it. Still, the use of profile-compatibility-indicator
>> does not harm session negotiation. The reason for not achieving
>> interoperability is another one.
>> 
>> 3) Offerer includes profile A and sets the flag related to profile
>> B in the profile-compatibility-indicator (Offerer has to be aware
>> of the intersection of both profiles). Answerer ist also aware of
>> profile A and the intersection with B and can accept the offer and
>> answer with a stream to the offerer that can be decoded. In this
>> case profile-compatibility-indicator helps for achieving 
>> interoperability.
> 
> Case 1: Offer: a=recvonly: In this case, there was no point of
> offering the profile-compatibility-indicator, as that would only
> unnecessary restrict the answerer in its sending of the stream.
> 
> [YK]: There can be still be a point to offer
> profile-compatibility-indicator in this case. For example, it is
> possible that the decoder of the offerer only supports the common
> tools of profiles A and B, while it does not support either full
> profile A or full profile B. Think about similar situation for the
> AVC Constrained Baseline profile, which contains the common tools of
> the Baseline profile and the Main profile, and was defined after the
> definitions of the Baseline profile and the Main profile, and uses
> the same profile_idc as the Baseline profile. Similar situations
> could happen for HEVC. Note that this can also happen when the
> direction attribute has other values.

Then the decoder is not compliant with any profile of A and B, if it
only supports the common subset. But, at least we share the
interpretation of what the profile-compatibility-indicator means in this
case. The answer's encoder MUST be capable of producing a bitstream that
fulfils the resulting constraint of producing a bitstream that is
decodable with a decoder compliant according to either profile A or B or
both.

>From my perspective and in the interest of efficient profile handling
and interoperability, if we care about the intersection of profiles A
and B, then define profile C that is that intersection. Don't play
around with this mechanism to achieve this.

Thus, I still think this is unnecessarily constraint on the answerer's
encoder.

> 
> Case 2: Offer: a=sendonly So in this case the offerer can include the
> However, the receiver if it accepts the RTP payload type, it promises
> to accept any stream that matches the RTP payload type. Thus, if one
> sends an additional stream under this RTP payload type what are the
> promise. It can't receive any stream fulfilling the profile-id
> parameter, it MUST also follow the constraints of the
> profile-compatibility-indicator. So what happens if that second
> encoder does not know about this profile. Then you end up in a
> incompatibility. Sure, with tight enforcement of that any stream you
> are being sent must fulfil to have the same
> [YK]: I see your point. In this case, if the offerer both profile A
> and profile B (by profile-id together with
> profile-compatibility-indicator), while the answerer understands
> profile A but does not understand profile B, the answerer can
> actually still accept the offer for the video communication from the
> offerer-to-answerer direction. However, for the other direction
> (answerer-to-offerer direction), the answerer does not know which
> tools to disable when encoding the video, there is no way to ensure
> interoperable video media in this direction. So, this can only work
> for single-direction communication, or declarative mode.

Correct!

> 
> Case 3: Offer: a=sendrecv Answerer: a=recvonly
> 
> Equal to above
> 
> Case 4: Offer: a=sendrecv Answerer: a=sendonly The answerer will not
> accept to receive the stream from Offerer, thus the answerer can
> accept the payload type, given that it can provided a stream matching
> the RTP payload type. However, as in this scenario the answerer don't
> support the offered profile-id, only a another supported profile.
> Thus, it doesn't fulfil the payload type and must reject it.
> 
> [YK]: I don't understand the first part above (thus the answerer can
> accept the payload type ...) - maybe there is a typo. But you second
> part seems to be aligned with my understanding, which is as follows
> (just to make the example clearer such that we are talking about the
> same thing). In this case, if the offerer indicates profile A and
> profile B (by profile-id together with
> profile-compatibility-indicator), while the answerer understands
> profile A but does not understand profile B, since the answerer does
> not know which tools to disable when encoding the video, there is no
> way to ensure interoperable video media in this direction, thus it
> has to reject the payload type.

Correct!

> 
> Case 5: Offer: a=sendrecv Answerer: a=sendrecv
> 
> that it can't send the offerer a stream matching the RTP payload
> type, see just above with answerer:sendonly.
> 
> Thus, I would say that it provides some benefit in the cases the
> offerer will only send the stream to answerer, but in no other cases.
> Also, this might give the offerer the wrong impression, and if one
> tries to modify the session to bi-directional streams, then this
> fails per the other options.
> 
> [YK]: The conclusion here should be slightly changed per Case 1. One
> optimization we can do here is to relax the constraint (that both
> profile_id and profile-compatibility-indicator must remain unchanged
> in offer and answer) a bit to enable Cases 2 and 3 (for
> single-direction from offerer to answerer, or declarative usage). For
> other cases, the current constraint applies (or as I said earlier,
> profile_id and profile-compatibility-indicator in both offer and
> answer must indicate the same set of encoding tools supported).

Yes, I only looked at the implications of one direction as they was
sufficient to reject the payload type. In the other direction it learns
that it can receive it. But, they are still not compatible for
bi-directional communication.

Sorry, I don't understand your proposal for relaxing the constraints.

> compatible profile that the encoder can send, i.e. each profile
> marked in profile-compatibility-indicator will get its own RTP
> payload type:
> 
> Case 6: Offer: a=recvonly: Given that the decoder has the capability
> for all profiles the encoder could produce, the compatibility would
> be maximized. However, this is mostly irrelvant in this comparision.
> 
> [YK]: See discussions above for Case 1. This case can actually be
> split into two cases. Case 6.1: if the decoder does fully support
> each of the multiple profiles, then using one payload type for each
> of these profiles is OK. Case 6.2: if the decoder only supports the
> common tools of a list of profiles but not fully for each of the
> profiles, then using one payload type for each of these profiles is
> NOT OK.

Yes, but that means going into sub-profiles, rather than assuming that
the decoder is compliant with particular profiles. I don't think we
should go into that discussion and the resulting issues.

> 
> Case 7: Offer: a=sendonly If the offerer includes both the intended
> profile and one for the compatible, then the answer rejects the
> intended one and accepts the PT for the compatible one. Things work,
> given that the answerer accepts a bitstream with profile-Id =
> intended and profile-compatibility-indicator=compatible one.
> 
> Case 8: Offer: a=sendrecv Answerer: a=recvonly Equal to above with
> the additional benefit that the offerer knows which profile the
> answerer supports to receive and likely send.
> 
> Case 9: Offer: a=sendrecv Answerer: a=sendonly To be able to offer
> this, the offerer must also be able to receive the compatible
> profile(s). If it doesn't then it can't offer multiple profiles and
> then we anyway have compatibility rejection.
> 
> Case 10: Offer: a=sendrecv Answerer: a=sendrecv To be able to offer
> this, the offerer must also be able to receive the compatible
> profile(s). If not we have failure as there is no compatible profile
> both can produce. If it can we have interoperability, and answerer
> indicates that it can only produce the other profile.
> 
> You might now better understand why I bring this up. It mostly fails
> or appear to have none intentional restrictions.
> 
> So I will admit that profile-compatibility-indicator provides a
> benefit if, and only if one offers sendonly streams, or ends up in
> sendonly from the party offering the profile-compatibility-indicator.
> This appears to indicate that having this as a sender constraint only
> would reduce the cases where it restricts. However, in this case one
> do loose the possibility to negotiate for a central mixer case, that
> you must produce a stream that fulfils the subset of these profiles.
> I do note that it appear necessary to make it clear that one MUST
> accept a video bitstream as long as its profile-id or
> profile-compatibility-indicator indicates the profile one has
> negotiated for the payload type. Is that clear in the HEVC
> specification? Does it need to be repeated in the payload format to
> prevent packetizers and others to make this error if they look
> However, if we are getting into this detail of negotiation, then I
> would suggest that we create one profile-compatibility-indicator
> signalling attribute per send and receive direction. So that one can
> have senders constraints without the receive constraints.
> 
> [YK]: I hope with my above comments, particularly those under Case 1,
> some of the conclusions above would be adjusted. And also please
> check my suggestion in the comments under Case 5.

To repeat, I don't think we shall support sub-profile support. Secondly,
I don't understand what you meant in case 5.

>> profile-compatibility-indicator could be harmful. Lets assume
>> profile A is a superset of profile B. The answerer could offer
>> profile A and set the flag corresponding to profile B in 
>> decode and encode such a stream would have to reject the payload
>> type if it is not aware of what means to have a stream being
>> compatible to profile A due to not knowing profile A. I think this
>> is what you are concern about right?
> 
> profile-compatibility-indicator. This is only relevant if one is
> forced to use the profile-compatibility-indicator as send direction
> constraint in an encoder. In a reception only case, one can look for
> just a profile one supports. Which further speaks for splitting the
> send and receiver constrints.
> 
>> profile B) and all profiles that are supersets of it are marked in
>> the profile-compatibility-indicator. All other cases of profiles
>> not being supersets and being marked in
>> profile-compatibility-indicator would be either example 2 or 3.
>> 
>> further the profile-id should indicate such a profile and
>> profile-compatibility-indicator should not be included. Obviously
>> this only applies for O/A and not for declarative session 
>> description.
> 
> [YK]: I think I now do see your points in Case 2 and 3, and as I said
> above, we can do some optimization (see again my suggestion above
> under Case 2). However, considering my above comments, particularly
> those under Case 1, I hope we can have a more comprehensive, and
> on-the-same-page understanding on this issue.
> 

Yes, I think we have a better understanding. Based on my understanding I
want to further clarify what we should consider to do here.

A) We should not go into the sub-profile discussion and dealing with the
intersection of profiles as receiver capability.

B) Unless we see a strong need to actually require encoders to produce
bit-streams that are compliant with multiple profiles, we can actually
remove the receiver direction interpretation of
profile-compatibility-indicator.

C) We define profile-compatibility-indicator as being purely a sender
direction constraint. Assuming that we can mandate that a profile B
decoder that accepts a RTP payload type that says profile-ID=A and
profile-compatibility-indicator:B=Yes is capable of receiving that
bitstream that enables at least the usage in unidirectional cases.

This leaves us with a question if the answerer can send the offerer a
bitstream that is Profile-ID=B and profile-compatibility-indicator:A=yes
using this offered payload type? For that to work, we will need a
general requirement that decoders can handle incoming bit-streams with
any profile-id value as long as the profile-compatibility-indicator in
the bistream is marked as compliant to one profile the decoder supports.
That is not obvious for me.

Cheers

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 6 May 2014 14:56:58 +0000

Magnus,

I think the only key different understanding between us now is the following: Whether to assume that a decoder does not fully support any profile indicated by profile-id (what you called "sub-profile", e.g. only support the common tools of two or more profiles).

Once we converge here, it should be easy to derive that text changes should be made. Thus, let me focus on this in this email.

Your opinion is obvious and strong - don't go into that direction.

My opinion is strong too - we do need to go into that direction, because of the following reasons:

- The ITU-T/MPEG control how to use the bits of profile-id and profile-compatibility-indicator in the video coding specifications. It is simply possible that a new profile can be defined and indicated without using a new value of profile-id, but indicated as conforming to two profiles. This happened for AVC for the Constrained Baseline profile as I explained. Also, in the recently finalized HEVC Range Extensions spec (see here: http://phenix.int-evry.fr/jct/doc_end_user/documents/17_Valencia/wg11/JCTVC-Q1005-v4.zip), 19 new profiles have been defined, indicated by only one value of profile-id and some values of bits of the interop-constraints field (Note, not the profile-compatibility-indicator field).

- Application standards organizations, e.g. 3GPP, may require "sub-profile" capability for decoders. Again, AVC Constrained Baseline profile is an example - Magnus you should know this much better than many others.

Thus, when defining the HEVC RTP payload here, I don't think we can ignore the above. Rather, we must go into the direction of "sub-profile".

BR, YK

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Tuesday, May 06, 2014 2:56 AM
To: Wang, Ye-Kui; Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

On 2014-05-06 02:38, Wang, Ye-Kui wrote:
>> profile-compatibility-indicator. Answerer cannot accept profile A so 
>> they cannot achieve interoperability.
>> 
>> in the profile-compatibility-indicator (Offerer has to be aware of 
>> the intersection of both profiles). Answerer ist not aware of profile 
>> A and does not know which are the tools that lay in the intersection, 
>> so again interoperability cannot be achieved, since the answerer 
>> cannot constraint the sent stream so that the offerer can decode it. 
>> in the profile-compatibility-indicator (Offerer has to be aware of 
>> the intersection of both profiles). Answerer ist also aware of 
>> answer with a stream to the offerer that can be decoded. In this case 
>> profile-compatibility-indicator helps for achieving interoperability.
> 
> profile A or full profile B. Think about similar situation for the AVC 
> Constrained Baseline profile, which contains the common tools of the 
> Baseline profile and the Main profile, and was defined after the 
> definitions of the Baseline profile and the Main profile, and uses the 
> same profile_idc as the Baseline profile. Similar situations could 
> happen for HEVC. Note that this can also happen when the direction 
> attribute has other values.

Then the decoder is not compliant with any profile of A and B, if it only supports the common subset. But, at least we share the interpretation of what the profile-compatibility-indicator means in this case. The answer's encoder MUST be capable of producing a bitstream that fulfils the resulting constraint of producing a bitstream that is decodable with a decoder compliant according to either profile A or B or both.

>From my perspective and in the interest of efficient profile handling and interoperability, if we care about the intersection of profiles A and B, then define profile C that is that intersection. Don't play around with this mechanism to achieve this.

Thus, I still think this is unnecessarily constraint on the answerer's encoder.

> profile A but does not understand profile B, the answerer can actually 
> still accept the offer for the video communication from the 

Correct!

> way to ensure interoperable video media in this direction, thus it has 
> to reject the payload type.

Correct!

> tries to modify the session to bi-directional streams, then this fails 
> per the other options.
> 

Yes, I only looked at the implications of one direction as they was sufficient to reject the payload type. In the other direction it learns that it can receive it. But, they are still not compatible for bi-directional communication.

> for all profiles the encoder could produce, the compatibility would be 
> maximized. However, this is mostly irrelvant in this comparision.
> 
> split into two cases. Case 6.1: if the decoder does fully support each 
> of the multiple profiles, then using one payload type for each of 
> these profiles is OK. Case 6.2: if the decoder only supports the 

Yes, but that means going into sub-profiles, rather than assuming that the decoder is compliant with particular profiles. I don't think we should go into that discussion and the resulting issues.

> given that the answerer accepts a bitstream with profile-Id = intended 
> and profile-compatibility-indicator=compatible one.
> 
> Case 8: Offer: a=sendrecv Answerer: a=recvonly Equal to above with the 
> additional benefit that the offerer knows which profile the answerer 
> supports to receive and likely send.
> 
> some of the conclusions above would be adjusted. And also please check 
> my suggestion in the comments under Case 5.

To repeat, I don't think we shall support sub-profile support. Secondly, I don't understand what you meant in case 5.

>> the profile-compatibility-indicator. All other cases of profiles not 
>> being supersets and being marked in profile-compatibility-indicator 
>> would be either example 2 or 3.
>> 

Yes, I think we have a better understanding. Based on my understanding I want to further clarify what we should consider to do here.

A) We should not go into the sub-profile discussion and dealing with the intersection of profiles as receiver capability.

B) Unless we see a strong need to actually require encoders to produce bit-streams that are compliant with multiple profiles, we can actually remove the receiver direction interpretation of profile-compatibility-indicator.

C) We define profile-compatibility-indicator as being purely a sender direction constraint. Assuming that we can mandate that a profile B decoder that accepts a RTP payload type that says profile-ID=A and profile-compatibility-indicator:B=Yes is capable of receiving that bitstream that enables at least the usage in unidirectional cases.

This leaves us with a question if the answerer can send the offerer a bitstream that is Profile-ID=B and profile-compatibility-indicator:A=yes
using this offered payload type? For that to work, we will need a general requirement that decoders can handle incoming bit-streams with any profile-id value as long as the profile-compatibility-indicator in the bistream is marked as compliant to one profile the decoder supports.
That is not obvious for me.

Cheers

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 6 May 2014 14:59:22 +0000

Typo correction: " Whether to assume that a decoder does not fully support ..." -> " Whether to assume ***it is possible*** that a decoder does not fully support ..."

BR, YK

-----Original Message-----
From: Wang, Ye-Kui 
Sent: Tuesday, May 06, 2014 7:57 AM
To: 'Magnus Westerlund'; Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: RE: [payload] Comments on draft-ietf-payload-rtp-h265-02

Magnus,

BR, YK

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]]
Sent: Tuesday, May 06, 2014 2:56 AM
To: Wang, Ye-Kui; Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Correct!

Correct!

Cheers

Magnus Westerlund

---

From: Yago Sanchez <[email-masked]>
Date: Wed, 7 May 2014 10:42:13 +0200

Hi Magnus,

If we go this way we would be removing the purpose of using the VPS for session negotiation.

Deprecating the use of recv-sub-layer-id to stream selection and not using it for capability indication does not make much sense to me. Why should an answerer choose a lower sub-layer if it is claiming that it could get the highest one? There might be other reasons but the main one I can think of is because it cannot receive the highest sub-layer due to its capabilities. Thus, even though recv-sub-layer-id was used for stream selection, the offerer could reconfigure the encoder leading to the stream selection not being persistent, unless we stated somewhere that this is not allowed.

Note also that right now the draft says that when recv-sub-layer-id is present the media configuration parameters MUST NOT be present, which means we cannot use recv-sub-layer-id for selection and add the level-id for level downgrade for capability indication. When an offerer gets the recv-sub-layer-id it could easily understand the capabilities of the answerer since it sent the sprop-vps but I see your concern for middleboxes which have to keep track of what was sent by the offerer. I did not see that as an issue but if you do, one potential way of solving it is to remove the part that says that media configuration parameters MUST NOT be present. Thus level downgrade could be accomplished while here we need a MUST for the level-id being equal to the level indicated in the VPS for the selected sub-layer. However, other parameters such as profile-id could be potentially different for different sub-layers, which means that the symmetry requirement of configuration parameters should be relaxed conditioned on whether the recv-sub-layer-id is present or not.

Best regards,
Yago

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 8 May 2014 14:38:37 +0200

Hi,

Please see inline.

On 2014-05-07 10:42, Yago Sanchez wrote:
> Deprecating the use of recv-sub-layer-id to stream selection and not
> using it for capability indication does not make much sense to me. Why
> should an answerer choose a lower sub-layer if it is claiming that it
> could get the highest one? There might be other reasons but the main one
> I can think of is because it cannot receive the highest sub-layer due to
> its capabilities. Thus, even though recv-sub-layer-id was used for
> stream selection, the offerer could reconfigure the encoder leading to
> the stream selection not being persistent, unless we stated somewhere
> that this is not allowed.

I have to say that I am not a fan of performing selection and capability
determination based on having to parse the VPS. The reason is that it
makes the work of any intermediary including policy functions like these
that exists in IMS more difficult to implement. Not only need to to
understand the parameters in the SDP, they also need to include a HEVC
VPS parser, and it needs to keep the offer VPS as state to determine
what the answer means, as this is only a reference into that structure.

> 
> Note also that right now the draft says that when recv-sub-layer-id is
> present the media configuration parameters MUST NOT be present, which
> means we cannot use recv-sub-layer-id for selection and add the level-id
> for level downgrade for capability indication. When an offerer gets the
> recv-sub-layer-id it could easily understand the capabilities of the
> answerer since it sent the sprop-vps but I see your concern for
> middleboxes which have to keep track of what was sent by the offerer. I
> did not see that as an issue but if you do, one potential way of solving
> it is to remove the part that says that media configuration parameters
> MUST NOT be present. Thus level downgrade could be accomplished while
> here we need a MUST for the level-id being equal to the level indicated
> in the VPS for the selected sub-layer. However, other parameters such as
> profile-id could be potentially different for different sub-layers,
> which means that the symmetry requirement of configuration parameters
> should be relaxed conditioned on whether the recv-sub-layer-id is
> present or not.

Yes, I think my concerns are lessen by including the answer parameters
that matches what the recv-sub-layer-id layer requires. But, if that is
done I still wonder what the advantage is of using the VPS for
capability determination. Is it that the existing tools with profile and
layers aren't fine grained enough. Or is it completely focused on the
need to negotiate layer information. Something that will need a more
complete (and likely complex) solution ones the scalable HEVC extensions
are defined.

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 13 May 2014 16:30:41 +0200

Ye-Kui and WG,

Please see inline. I think we need input from more people into this
question to make progress.

On 2014-05-06 16:56, Wang, Ye-Kui wrote:
> Magnus,
> 
> I think the only key different understanding between us now is the
> following: Whether to assume that a decoder does not fully support
> any profile indicated by profile-id (what you called "sub-profile",
> e.g. only support the common tools of two or more profiles).
> 
> Once we converge here, it should be easy to derive that text changes
> should be made. Thus, let me focus on this in this email.
> 
> - The ITU-T/MPEG control how to use the bits of profile-id and
> profile-compatibility-indicator in the video coding specifications.
> It is simply possible that a new profile can be defined and indicated
> without using a new value of profile-id, but indicated as conforming
> to two profiles. This happened for AVC for the Constrained Baseline
> profile as I explained. Also, in the recently finalized HEVC Range
> Extensions spec (see here:
> http://phenix.int-evry.fr/jct/doc_end_user/documents/17_Valencia/wg11/JCTVC-Q1005-v4.zip),
> 19 new profiles have been defined, indicated by only one value of
> profile-id and some values of bits of the interop-constraints field
> (Note, not the profile-compatibility-indicator field).

Yes this may happen, but jumping onto this now, do in fact create a tool
where anyone can attempt to create new interoperability points by
selecting an intersection between profiles. That doesn't benefit
interoperability, it instead fractures the market and works against the
purpose of the profiles in the first place.

Secondly, in relation to H.264 constrained baseline, that was created
using constraint bits, a concept that already is part of the
specification, it might be defined as the intersection of two profiles,
but the signalling is done according to one profile with a specific bit
set. Just like the HEVC Range Extension profiles.

However, to me it appears that when one selected the structure for HEVC
Range Extensions one forgot about the profile-compatibility-indicator as
it is not possible to signal compatibility across the main directions in
the range extension profiles. The main directions are within themselves
nicely onion shaped, but the compatibility between them does not appear
to have been well considered.

> 
> - Application standards organizations, e.g. 3GPP, may require
> "sub-profile" capability for decoders. Again, AVC Constrained
> Baseline profile is an example - Magnus you should know this much
> better than many others.

Yes, however I think it is usually bad for interoperability. And HEVC
does not appear to be in the same position that H.264 was when it was
just finished. Thus, I expect the issues are being different.

> 
> Thus, when defining the HEVC RTP payload here, I don't think we can
> ignore the above. Rather, we must go into the direction of
> "sub-profile".

I understand your arguments. What is required is highly dependent on
what will be done with future profiling of HEVC. I see enabling the
possibility to intersect any choice of profiles do create an potential
for fracturing the market and reducing the interoperability of HEVC.

Cheers

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 13 May 2014 16:57:34 +0000

Magnus,

I understand your arguments too. Your main worry seems to be about the HEVC profiling fracturing the market and reducing the interoperability of HEVC, which I do share. However, the key point here is that, that is not up to us, it is controlled by other organizations. If we go ahead to disallow the support of that in the HEVC payload format, that does not mean we would successfully prevent that from happening in MPEG/ITU-T and other application standard organizations - and if that happens, then an extension to the payload format would need to be specified, which might introduce more IOP issues even on the RTP encapsulation level for HEVC. 

I fully agree that some more opinions from others would be very helpful.

Please, whoever cares about this draft and cares about using HEVC with RTP, say your opinion. We really need to have this document finalized soon for HEVC deployment!!!

See some other detailed replies inline below too.

BR, YK

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Tuesday, May 13, 2014 7:31 AM
To: Wang, Ye-Kui; Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Ye-Kui and WG,

Please see inline. I think we need input from more people into this question to make progress.

On 2014-05-06 16:56, Wang, Ye-Kui wrote:
> following: Whether to assume that a decoder does not fully support any 
> profile indicated by profile-id (what you called "sub-profile", e.g. 
> only support the common tools of two or more profiles).
> 

Yes this may happen, but jumping onto this now, do in fact create a tool where anyone can attempt to create new interoperability points by selecting an intersection between profiles. That doesn't benefit interoperability, it instead fractures the market and works against the purpose of the profiles in the first place.

[YK]: Agreed - but again that is not up to us, it is controlled by other SDOs (which are in turn driven by the markets).

Secondly, in relation to H.264 constrained baseline, that was created using constraint bits, a concept that already is part of the specification, it might be defined as the intersection of two profiles, but the signalling is done according to one profile with a specific bit set. Just like the HEVC Range Extension profiles.

[YK]: Why the signalling is different? Herein you also use a profile-id value with a specific bit set.

However, to me it appears that when one selected the structure for HEVC Range Extensions one forgot about the profile-compatibility-indicator as it is not possible to signal compatibility across the main directions in the range extension profiles. The main directions are within themselves nicely onion shaped, but the compatibility between them does not appear to have been well considered.

[YK]: I was not involved in the definition of the structure for HEVC Range Extensions - so no comment here.

> "sub-profile" capability for decoders. Again, AVC Constrained Baseline 
> profile is an example - Magnus you should know this much better than 
> many others.

Yes, however I think it is usually bad for interoperability. And HEVC does not appear to be in the same position that H.264 was when it was just finished. Thus, I expect the issues are being different.

[YK]: True - but on the other hand, that was driven by the market too, and they wanted that for interoperability for that particular capability. Here the issue is not about what was defined already in HEVC, but what could be done in the future. When RFC 3984 was finalized, the Constrained Baseline profile was not available yet. So, right now, the situation is the same. Unless we can surely know similar things won't happen, we should follow the approach in RFC 3984, i.e. to allow for such intersection of two more profiles. Nobody can be sure right now for what could happen in the future, at least for this issue.

> ignore the above. Rather, we must go into the direction of 

I understand your arguments. What is required is highly dependent on what will be done with future profiling of HEVC. I see enabling the possibility to intersect any choice of profiles do create an potential for fracturing the market and reducing the interoperability of HEVC.

[YK]: Again, agree in general, but again that is not controlled by us.

Cheers

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 13 May 2014 17:11:30 +0000

So we have the following options:

A: To keep what we have now. This way, the SDP answer message can be slim, but it does require entities involved in O/A to parse some parts of the VPS.

B: To completely remove the use of VPS for session negotiation in O/A. This way, the SDP answer message would be fatter compared to A, but it does require entities involved in O/A to parse any parts of the VPS.

C: In between A and B, by allowing for the use of the VPS for session negotiation in O/A, but require the key parts like profile, level, and other Configuration parameters to be present in the answer SDP too. This way, the SDP answer would probably be even a litter bit fatter than B, and it entities involved in O/A may or may not parse the VPS.

I personally prefer A, which is aligned with what we have in RFC 6190 (the SVC payload format), and would be directly applicable for SHVC (the scalable extension of HEVC) and MV-HEVC (the multiview extension of HEVC) as well.

Again, I'd like to ask those who care about this draft and about deploying HEVC over RTP to express their opinions, as really the market is deploying HEVC already and we should try to get this document finalized as soon as possible. Thanks!!!

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: Thursday, May 08, 2014 5:39 AM
To: Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02- on recv-sub-layer-id

Hi,

Please see inline.

On 2014-05-07 10:42, Yago Sanchez wrote:
> could get the highest one? There might be other reasons but the main 
> one I can think of is because it cannot receive the highest sub-layer 
> due to its capabilities. Thus, even though recv-sub-layer-id was used 
> for stream selection, the offerer could reconfigure the encoder 
> leading to the stream selection not being persistent, unless we stated 
> somewhere that this is not allowed.

I have to say that I am not a fan of performing selection and capability determination based on having to parse the VPS. The reason is that it makes the work of any intermediary including policy functions like these that exists in IMS more difficult to implement. Not only need to to understand the parameters in the SDP, they also need to include a HEVC VPS parser, and it needs to keep the offer VPS as state to determine what the answer means, as this is only a reference into that structure.

> means we cannot use recv-sub-layer-id for selection and add the 
> level-id for level downgrade for capability indication. When an 
> offerer gets the recv-sub-layer-id it could easily understand the 
> capabilities of the answerer since it sent the sprop-vps but I see 
> your concern for middleboxes which have to keep track of what was sent 
> by the offerer. I did not see that as an issue but if you do, one 
> potential way of solving it is to remove the part that says that media 
> configuration parameters MUST NOT be present. Thus level downgrade 
> could be accomplished while here we need a MUST for the level-id being 
> equal to the level indicated in the VPS for the selected sub-layer. 
> However, other parameters such as profile-id could be potentially 
> different for different sub-layers, which means that the symmetry 
> requirement of configuration parameters should be relaxed conditioned 
> on whether the recv-sub-layer-id is present or not.

Yes, I think my concerns are lessen by including the answer parameters that matches what the recv-sub-layer-id layer requires. But, if that is done I still wonder what the advantage is of using the VPS for capability determination. Is it that the existing tools with profile and layers aren't fine grained enough. Or is it completely focused on the need to negotiate layer information. Something that will need a more complete (and likely complex) solution ones the scalable HEVC extensions are defined.

Cheers

Magnus Westerlund

_______________________________________________
payload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/payload

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 13 May 2014 17:37:32 +0000

Hi,
I¹m one of the co-authors.
To make one thing clear from the outset: I¹m one who says that any and all
options of a given payload necessarily need always to be supported by a
RTP payload format.  Sometimes, it¹s IMO completely OK to forgo the
support of an odd mechanism that¹s not required or useful for the
application ³payload over the Internet using RTP².
That said, the case here is different.  What we have here is a request (by
Magnus) to narrow down the number of interoperability points negotiable
using O/A negotiation, by intentionally disabling a mechanism that was
expressly included for purposes just like this.  There was considerable
discussion in JCT-VC, MPEG, and VCEG about the subject.  I would say it
was one of the key drivers, but my view may be a bit one-sided.  However,
it was discussed more than once.  The front lines were often between the
hardware folks (who would like to have as few interop points as possible)
and the software people (who would prefer to have a finer granularity, to
better suit individual application needs).  The resulting compromise
necessarily made no one completely happy, but everyone could live with it.
Ye-Kui is right: it¹s not the IETF¹s job to rehash these profiling
architecture discussions/decisions.
Thanks for your consideration,
Stephan

On 13.5.14, 9:57, "Wang, Ye-Kui" <[email-masked]> wrote:

>Magnus,
>
>I understand your arguments too. Your main worry seems to be about the
>HEVC profiling fracturing the market and reducing the interoperability of
>HEVC, which I do share. However, the key point here is that, that is not
>up to us, it is controlled by other organizations. If we go ahead to
>disallow the support of that in the HEVC payload format, that does not
>mean we would successfully prevent that from happening in MPEG/ITU-T and
>other application standard organizations - and if that happens, then an
>extension to the payload format would need to be specified, which might
>introduce more IOP issues even on the RTP encapsulation level for HEVC.
>
>Please, whoever cares about this draft and cares about using HEVC with
>RTP, say your opinion. We really need to have this document finalized
>soon for HEVC deployment!!!
>
>[YK]: Agreed - but again that is not up to us, it is controlled by other
>SDOs (which are in turn driven by the markets).
>
>[YK]: True - but on the other hand, that was driven by the market too,
>and they wanted that for interoperability for that particular capability.
>Here the issue is not about what was defined already in HEVC, but what
>could be done in the future. When RFC 3984 was finalized, the Constrained
>Baseline profile was not available yet. So, right now, the situation is
>the same. Unless we can surely know similar things won't happen, we
>should follow the approach in RFC 3984, i.e. to allow for such
>intersection of two more profiles. Nobody can be sure right now for what
>could happen in the future, at least for this issue.
>
>> 
>I understand your arguments. What is required is highly dependent on what
>will be done with future profiling of HEVC. I see enabling the

---

From: Jonatan Samuelsson <[email-masked]>
Date: Fri, 16 May 2014 09:30:33 +0000

Dear Ye-Kui, Stefan, all,

Let me express my opinion on this topic (since you explicitly asked for more opinions). I do represent the same company as Magnus, but (in contrast to Magnus) I was also part of the discussions on how to define the profiles in HEVC version 1. I share Magnus' view that it would be very unfortunate if the payload format allows for definitions of sub-profiles using profile-compatibility-indicator. The purpose of the profile_compatibility_flags is very clear in the contribution that proposed them, JCTVC-I0499 (http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=5761):
"We need a labelling scheme that:
a)      allows the committee to introduce new profiles, and older systems be able to work out which bitstreams they can decode when those new profiles are fully subsets of older existing profiles, even when they do not recognize the new profile identifier;
b)      allows bitstreams which don't use all the features of any given profile, and are as a consequence compatible with more than one, to be labelled with so that it's possible to deduce all the profiles that the bitstream is compatible with (the ones that don't automatically follow from the nesting structure).
"
If there is an intersection between two profiles that does make sense and is useful to the industry it is reasonable to assume that ITU/ISO/IEC would define a new profile corresponding to that intersection (just as was done with Constrained Baseline Profile in h.264). And in HEVC, the profile_compatibility_flags provides a mechanism for introducing a new profile, with a new profile_idc while still being able to indicate conformance to one or more of the old profiles (a mechanism that did not exist in h.264).

However, it is worth noting that the way the Range Extension profiles have been defined in the second (soon to be released) version of HEVC, the functionality of profile_compatibility_flag is already broken. There are 19 new profiles in there but only one profile_idc is used to identify them. Thus it is not possible in the same bitstream to indicate that it conforms to two different of these 19 profiles.

Now to connect back to the example you mentioned, Ye-Kui, on May 5: "There can be still be a point to offer profile-compatibility-indicator in this case. For example, it is possible that the decoder of the offerer only supports the common tools of profiles A and B, while it does not support either full profile A or full profile B.

1.       I don't think this is desirable.

2.       Within the profiles defined in "format range extension" (~90% of the profiles defined so far) this is not possible.

3.       When I read the latest HEVC payload format draft, I don't find anything supporting that kind of usage of profile-compatibility-indicator. On the contrary:

         A decoder conforming to a certain profile may

         be able to decode bitstreams conforming to other profiles.

         The profile-compatibility-indicator provides exact information

         of the ability of a decoder conforming to a certain profile to

         decode bitstreams conforming to another profile.  More

         concretely, if the profile compatibility flag corresponding to

         the profile a decoder conforms to is set, then the decoder is

         able to decode any bitstream with the flag set, irrespective

         of the profile the bitstream conforms to (provided that the

         decoder supports the highest level of the bitstream).

I can see the benefit in using the interop-constraints to express constraints that applies to well-defined profiles (both from a decoder and encoder point of view). However, using profile-compatibility-indicator to let decoders create ad-hoc on the fly "sub-profiles" of an arbitrary number of defined profiles serves IMO no purpose and is simply a misuse of the profile_compatibility_flags.

Best Regards Jonatan

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Stephan Wenger
Sent: den 13 maj 2014 19:38
To: Wang, Ye-Kui; Magnus Westerlund; Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi,

To make one thing clear from the outset: I¹m one who says that any and all options of a given payload necessarily need always to be supported by a RTP payload format.  Sometimes, it¹s IMO completely OK to forgo the support of an odd mechanism that¹s not required or useful for the application ³payload over the Internet using RTP².

Magnus) to narrow down the number of interoperability points negotiable using O/A negotiation, by intentionally disabling a mechanism that was expressly included for purposes just like this.  There was considerable discussion in JCT-VC, MPEG, and VCEG about the subject.  I would say it was one of the key drivers, but my view may be a bit one-sided.  However, it was discussed more than once.  The front lines were often between the hardware folks (who would like to have as few interop points as possible) and the software people (who would prefer to have a finer granularity, to better suit individual application needs).  The resulting compromise necessarily made no one completely happy, but everyone could live with it.

Ye-Kui is right: it¹s not the IETF¹s job to rehash these profiling architecture discussions/decisions.

Stephan

>HEVC profiling fracturing the market and reducing the interoperability

>of HEVC, which I do share. However, the key point here is that, that is

>not up to us, it is controlled by other organizations. If we go ahead

>to disallow the support of that in the HEVC payload format, that does

>not mean we would successfully prevent that from happening in

>MPEG/ITU-T and other application standard organizations - and if that

>happens, then an extension to the payload format would need to be

>specified, which might introduce more IOP issues even on the RTP encapsulation level for HEVC.

>Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

>> any profile indicated by profile-id (what you called "sub-profile", e.g.

>> My opinion is strong too - we do need to go into that direction,

>> because of the following reasons:

>> http://phenix.int-evry.fr/jct/doc_end_user/documents/17_Valencia/wg11

>> /

>> JCTVC-Q1005-v4.zip),

>Yes this may happen, but jumping onto this now, do in fact create a

>tool where anyone can attempt to create new interoperability points by

>[YK]: Agreed - but again that is not up to us, it is controlled by

>other SDOs (which are in turn driven by the markets).

>[YK]: Why the signalling is different? Herein you also use a profile-id

>value with a specific bit set.

>Range Extensions one forgot about the profile-compatibility-indicator

>as it is not possible to signal compatibility across the main

>directions in the range extension profiles. The main directions are

>within themselves nicely onion shaped, but the compatibility between

>them does not appear to have been well considered.

>[YK]: I was not involved in the definition of the structure for HEVC

>Range Extensions - so no comment here.

>could be done in the future. When RFC 3984 was finalized, the

>Constrained Baseline profile was not available yet. So, right now, the

>situation is the same. Unless we can surely know similar things won't

>happen, we should follow the approach in RFC 3984, i.e. to allow for

>such intersection of two more profiles. Nobody can be sure right now

>for what could happen in the future, at least for this issue.

>SE-164 80 Stockholm, Sweden | mailto: [email-masked]<mailto:[email-masked]>

>[email-masked]<mailto:[email-masked]>

payload mailing list

[email-masked]<mailto:[email-masked]>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 16 May 2014 18:20:01 +0000

Hi Jonatan,

Thanks for responding to my request for comments - I really appreciate (even though you are not supporting my opinion :)).

Frankly, I think the observations you pointed out below do not support the opinion you expressed (See more detailed replies inline below.).

What are being discussed here, as Stephan accurately pointed out, is whether to narrow down the number of interoperability points negotiable using O/A negotiation, by intentionally disabling a mechanism that was expressly included for purposes that might happen and that have happened.  To do (the narrowing down) or not to do, that is the question. And the key point to dertermine the answer to the question is whether THAT (the intersection way of profiling) might happen. Unless it is fairly sure that THAT won't happen, we ought to provide the support of THAT in the payload format.

The facts that THAT did happened for H.264 and that whether it would happen is not controlled by IETF, but by other organizations, more than one of them, I'd say that we are far from being fairly sure that THAT won't happen. Again, if the narrowing is done and if THAT occurs, then an extension to the payload format needs to be specified to add the support of this case and the design would be the same as what we have now, except that it introduces new IOP issue on the payload format level (due to two incompatible versions of the payload format), and if the narrowing is not done and if THAT does not occur, all the hypothetical issues won't happen anyway.

BR, YK

From: Jonatan Samuelsson [mailto:[email-masked]]
Sent: Friday, May 16, 2014 2:31 AM
To: Stephan Wenger; Wang, Ye-Kui; Magnus Westerlund; Yago Sanchez
Cc: [email-masked]; [email-masked]
Subject: RE: [payload] Comments on draft-ietf-payload-rtp-h265-02

[YK]: Item b above talking about exactly encoding a bitstream using an intersection of multiple profiles, which is exactly the point we are taking about.

If there is an intersection between two profiles that does make sense and is useful to the industry it is reasonable to assume that ITU/ISO/IEC would define a new profile corresponding to that intersection (just as was done with Constrained Baseline Profile in h.264).

[YK]: Defining a new profile does not necessarily means using of a new value of the profile-id for indicating the new profile, just as was done with the Constrained Baseline profile of H.264.

And in HEVC, the profile_compatibility_flags provides a mechanism for introducing a new profile, with a new profile_idc while still being able to indicate conformance to one or more of the old profiles (a mechanism that did not exist in h.264).

[YK]: The mechanism of profile_compatibility_flags is very similar to what H.264 has, with the only difference been that in HEVC there are 32 of these flags and 5-bit of profile-id, while in H.264 there are less bits of these flags and 8 bits of profile-id.

[YK]: Which functionality of  profile_compatibility_flag as specified in the HEVC spec is broken? I guess it is just the functionality of profile_compatibility_flag in your understanding is broken :) We should not assume anything more than said in the spec. I don't see anything written in the spec is broken, even though it was a little surprise to me too when seeing the definition of the Range Extension profiles (as I did not expect to use the interop-constraints bits for profile definition either). The fact that the interop-constraints bits have been used in profile definition proves more that the way of profile signalling is not controlled by us, and we should NOT preclude the support of such profile signalling. Fortunately, the mechanism we currently specify in the payload draft, where interop-constraints is one of the configuration parameters required for symmetric use in O/A, works with this.

1.       I don't think this is desirable.
[YK]: As I said earlier in replies to Magnus, I do share the concern either, but this is not up to us - let me repeat again.

2.       Within the profiles defined in "format range extension" (~90% of the profiles defined so far) this is not possible.
[YK]: There is no such principle or functionality that the example must be possible for all cases. As long as it is possible for any two of the 32 flags, the payload format should provide support. Otherwise, as I said earlier, we as the payload format designers are only introducing more IOP issues in the future once this is done (by other organizations).

[YK] Due to the newly introduced signalling of Range Extensions profiles using one profile-id value in combination with interop-constraints, the above description of profile-compatibility-indicator needs to be updated. I had planned to do this in the next version. But again, here we need to specify the text to align with whatever is specified in the HEVC specification.

[YK] See above.

Best Regards Jonatan

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Stephan Wenger
Sent: den 13 maj 2014 19:38
To: Wang, Ye-Kui; Magnus Westerlund; Yago Sanchez
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi,

Stephan

payload mailing list

---

From: Yago Sanchez <[email-masked]>
Date: Sat, 17 May 2014 12:06:28 +0200

Hi, Ye-Kui, Jonatan,all,

Just to make clear my opinion on this topic, I am here with Ye-Kui and Stephan as already expressed previously in this thread. Desirable or not, profile_compatibility_indicator allows for indicating intersection of profiles as expressed in the item b) in the contribution mentioned in the email below. Whether the intersections will get a profile_idc or not cannot be known beforehand and might not since we have profile_compatibility_flag for that purpose. Said that still I think the argument here is that if HEVC allows for it we should support it in the payload format.

Best regards,
Yago

On 16 May 2014, at 20:20, Wang, Ye-Kui <[email-masked]> wrote:

> Hi Jonatan,
>  
> Thanks for responding to my request for comments – I really appreciate (even though you are not supporting my opinion J).
>  
> What are being discussed here, as Stephan accurately pointed out, is whether to narrow down the number of interoperability points negotiable using O/A negotiation, by intentionally disabling a mechanism that was expressly included for purposes that might happen and that have happened.  To do (the narrowing down) or not to do, that is the question. And the key point to dertermine the answer to the question is whether THAT (the intersection way of profiling) might happen. Unless it is fairly sure that THAT won’t happen, we ought to provide the support of THAT in the payload format.
>  
> The facts that THAT did happened for H.264 and that whether it would happen is not controlled by IETF, but by other organizations, more than one of them, I’d say that we are far from being fairly sure that THAT won’t happen. Again, if the narrowing is done and if THAT occurs, then an extension to the payload format needs to be specified to add the support of this case and the design would be the same as what we have now, except that it introduces new IOP issue on the payload format level (due to two incompatible versions of the payload format), and if the narrowing is not done and if THAT does not occur, all the hypothetical issues won’t happen anyway.
>  
>  
> BR, YK
>  
> [YK]: Which functionality of  profile_compatibility_flag as specified in the HEVC spec is broken? I guess it is just the functionality of profile_compatibility_flag in your understanding is broken J We should not assume anything more than said in the spec. I don’t see anything written in the spec is broken, even though it was a little surprise to me too when seeing the definition of the Range Extension profiles (as I did not expect to use the interop-constraints bits for profile definition either). The fact that the interop-constraints bits have been used in profile definition proves more that the way of profile signalling is not controlled by us, and we should NOT preclude the support of such profile signalling. Fortunately, the mechanism we currently specify in the payload draft, where interop-constraints is one of the configuration parameters required for symmetric use in O/A, works with this.
>  
> Now to connect back to the example you mentioned, Ye-Kui, on May 5: “There can be still be a point to offer profile-compatibility-indicator in this case. For example, it is possible that the decoder of the offerer only supports the common tools of profiles A and B, while it does not support either full profile A or full profile B.
> 1.       I don’t think this is desirable.
> [YK]: As I said earlier in replies to Magnus, I do share the concern either, but this is not up to us – let me repeat again.
>  
> 2.       Within the profiles defined in “format range extension” (~90% of the profiles defined so far) this is not possible.
> 3.       When I read the latest HEVC payload format draft, I don’t find anything supporting that kind of usage of profile-compatibility-indicator. On the contrary:
>  
> I can see the benefit in using the interop-constraints to express constraints that applies to well-defined profiles (both from a decoder and encoder point of view). However, using profile-compatibility-indicator to let decoders create ad-hoc on the fly “sub-profiles” of an arbitrary number of defined profiles serves IMO no purpose and is simply a misuse of theprofile_compatibility_flags.
>  
> [YK] See above.
>  
> Best Regards Jonatan
>

---

From: Jonatan Samuelsson <[email-masked]>
Date: Mon, 19 May 2014 09:23:19 +0000

Hi Ye-Kui, Yago, all,

Just to make one thing clear; there is a HUGE difference between bitstreams that conforms to multiple profiles (which is what "item b)" below is about) and decoders that only supports intersections of multiple profiles. From an encoder implementers point of view it is a nightmare to handle all the different intersections that can occur. As soon as a new profile is defined in the HEVC specification an encoder implementer will not only have to add support for that profile but also for all the intersections with old profiles and all the intersections with the different intersections of old profiles. A complete explosion of operating points resulting in increased cost and higher risk of bugs and misinterpretations causing interoperability problems.

I think profile-compatibility-indicator is very useful as a bitstream property (sprop parameter). By using that, the same payload type can be accepted by answerer A (supporting profile pA) and answerer B (supporting profile pB).

Let me express the alternatives I see for profile-compatibility-indicator, in my order of preference:

1.       Making it a sprop parameter only (expressing a property of the bitstream)

2.       Making 2 parameters: one sprop parameter AND one receiver parameter that can be used to express that the decoder supports multiple (full) profiles (similar to the cited text below from the latest draft of the payload spec)

3.       Making it a single parameter used both for expressing that a bitstream conforms to multiple profiles and that a decoder only supports the intersection of all the indicated profiles (this is what you are proposing, right?).

I would strongly prefer if option 3 could be avoided for the above mentioned reason. If not, I guess we just have to ensure that all SDOs that uses the HEVC payload spec does not use this mechanism :). See some further discussion inline below.

Best Regards Jonatan

From: Yago Sanchez [mailto:[email-masked]]
Sent: den 17 maj 2014 12:06
To: Wang, Ye-Kui
Cc: Jonatan Samuelsson; Stephan Wenger; Magnus Westerlund; [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi, Ye-Kui, Jonatan,all,

[JS] As mentioned above; there is a HUGE difference between bitstreams that conforms to multiple profiles and decoders that only supports intersections of multiple profiles. The HEVC spec does not allow the latter. If we introduce the concept of decoders conforming to intersection of profiles in the HEVC payload format that would be something which does not have any correspondence in the HEVC spec.

Best regards,
Yago

On 16 May 2014, at 20:20, Wang, Ye-Kui <[email-masked]<mailto:[email-masked]>> wrote:

Hi Jonatan,

What are being discussed here, as Stephan accurately pointed out, is whether to narrow down the number of interoperability points negotiable using O/A negotiation, by intentionally disabling a mechanism that was expressly included for purposes that might happen and that have happened.  To do (the narrowing down) or not to do, that is the question. And the key point to dertermine the answer to the question is whether THAT (the intersection way of profiling) might happen. Unless it is fairly sure that THAT won't happen, we ought to provide the support of THAT in the payload format.
[JS] Given that the profiles in  HEVC version 1 are nicely "onion shaped" (which they were not in h.264) and given that there are 48 constraint flags (compared to the 8 in h.264) I am personally fairly sure THAT won't happen, but apparently you are not :).

The facts that THAT did happened for H.264 and that whether it would happen is not controlled by IETF, but by other organizations, more than one of them, I'd say that we are far from being fairly sure that THAT won't happen. Again, if the narrowing is done and if THAT occurs, then an extension to the payload format needs to be specified to add the support of this case and the design would be the same as what we have now, except that it introduces new IOP issue on the payload format level (due to two incompatible versions of the payload format), and if the narrowing is not done and if THAT does not occur, all the hypothetical issues won't happen anyway.
[JS] If THAT does occur sometime in the future, there are multiple options for how it could be handled in the payload format (depending on how specific the case for which it occurs is). I would much rather see some inconvenience for that specific case then, compared to this burdensome mechanism.

BR, YK

[YK]: The mechanism of profile_compatibility_flags is very similar to what H.264 has, with the only difference been that in HEVC there are 32 of these flags and 5-bit of profile-id, while in H.264 there are less bits of these flags and 8 bits of profile-id.
[JS] There is also the difference that the semantics of all the profile_compatibility_flags in HEVC are defined already in version 1. In h.264 there were only 3 bits defined in version 1 and the rest were reserved bits thus making them more comparable to the flags called "interop-constraints" in the HEVC payload draft.

[YK]: Which functionality of  profile_compatibility_flag as specified in the HEVC spec is broken? I guess it is just the functionality of profile_compatibility_flag in your understanding is broken :) We should not assume anything more than said in the spec. I don't see anything written in the spec is broken, even though it was a little surprise to me too when seeing the definition of the Range Extension profiles (as I did not expect to use the interop-constraints bits for profile definition either). The fact that the interop-constraints bits have been used in profile definition proves more that the way of profile signalling is not controlled by us, and we should NOT preclude the support of such profile signalling. Fortunately, the mechanism we currently specify in the payload draft, where interop-constraints is one of the configuration parameters required for symmetric use in O/A, works with this.
[JS] It appears broken to me:
general_profile_compatibility_flag[ j ] equal to 1, when general_profile_space is equal to 0, indicates that the CVS conforms to the profile indicated by general_profile_idc equal to i as specified in Annex A.
How should you interpret "the CVS conforms to the profile indicated by general_profile_idc equal to i" when i equals 4? Which is "the profile" in this case?

I can see the benefit in using the interop-constraints to express constraints that applies to well-defined profiles (both from a decoder and encoder point of view). However, using profile-compatibility-indicator to let decoders create ad-hoc on the fly "sub-profiles" of an arbitrary number of defined profiles serves IMO no purpose and is simply a misuse of theprofile_compatibility_flags.

[YK] See above.

Best Regards Jonatan

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Stephan Wenger
Sent: den 13 maj 2014 19:38
To: Wang, Ye-Kui; Magnus Westerlund; Yago Sanchez
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi,

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Mon, 19 May 2014 16:25:35 +0000

Hi Jonatan,

I think there is a big misunderstanding here. Just because the payload format supports the mechanism of a configuration of an intersection of more than one profile does not mean that encoders must be able to support all possible intersections. If you think so, please point out where is this implied.

BR, YK

From: Jonatan Samuelsson [mailto:[email-masked]]
Sent: Monday, May 19, 2014 2:23 AM
To: Yago Sanchez; Wang, Ye-Kui
Cc: Stephan Wenger; Magnus Westerlund; [email-masked]; [email-masked]
Subject: RE: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi Ye-Kui, Yago, all,

Best Regards Jonatan

From: Yago Sanchez [mailto:[email-masked]]
Sent: den 17 maj 2014 12:06
To: Wang, Ye-Kui
Cc: Jonatan Samuelsson; Stephan Wenger; Magnus Westerlund; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi, Ye-Kui, Jonatan,all,

Best regards,
Yago

Hi Jonatan,

BR, YK

[YK] See above.

Best Regards Jonatan

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Stephan Wenger
Sent: den 13 maj 2014 19:38
To: Wang, Ye-Kui; Magnus Westerlund; Yago Sanchez
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi,

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 28 May 2014 14:28:52 +0200

WG,

You can consider these issues closed from my side. I have been working
with the authors on the forthcoming -04. And all my major issues appear
resolved in that version. So there is no reason to not progress the
draft on my account.

cheers

Magnus Westerlund