Subject: Re: [avtext] Frame marking & VP9 SVC

From: Jonathan Lennox <[email-masked]>
Date: Wed, 5 Jul 2017 15:56:04 +0000

Sergio — the updated VP9 payload draft I submitted Monday includes a section describing its usage with frame marking.

Let me know what you think?  It changes the S and T fields to reflect the latest VP9 and frame marking drafts, but it does not include the P or U bits.

I think the intention is that frame marking’s B bit will provide a coarser-grain version of what P and U do (and the “don’t use it with weird structures” requirement will make that sufficient, though that may need to be made more specific).  What do you think?

On Jun 28, 2017, at 10:23 AM, Sergio Garcia Murillo <[email-masked]<mailto:[email-masked]>> wrote:

Yes, at least in VP9 SVC you can define how many temporal and how many spatial layers do you want simultaneously in the encoding. Note also, that VP9 supports quality layers as spatial layers without any resolution change, so the term "spatial layer" is used to represent both spatial and quality layers.

BR
Sergio
On 28/06/2017 16:18, Bernard Aboba wrote:
Sergio said:  "So, shouldn't the LID reference to the spatial layer ID only and omit the quality layer id completely? Also, the spatial layer id is 3 bits on that draft."

[BA] Question:  In VP9 (or AV1, for that matter) is it possible to use both spatial and quality scalability in the same encoding?

On Wed, Jun 28, 2017 at 5:35 AM, Sergio Garcia Murillo <[email-masked]<mailto:[email-masked]>> wrote:
Hi all again,

Did you had a chance to review the changes to add the VP9 SVC spatial layer information into the LID of the frame marking extension? (at https://github.com/murillo128/draft-berger-avtext-framemarking/commit/35cdb229108251f12bc1cdf77909df83a51d3994)

Also, FYI, I have submitted a CL to libwebrtc to support framemarking extension. Currently it supports VP8, H264 and non-SVC VP9, but I would love to tackle VP9 SVC as soon as we get an agreement on the changes to the LID:

https://codereview.webrtc.org/2954503002/

Best regards
Sergio

On 24/04/2017 11:28, Sergio Garcia Murillo wrote:
On 18/04/2017 17:40, Jonathan Lennox wrote:
One of the goals of Frame Marking is that an SFU can be agnostic to the details of the layering, and just treat layer IDs as opaque layer identifiers.  Putting non-layer bits into the LID field would break this goal.

At the IETF meeting in Chicago, both Bernard Aboba and I thought that frame marking might need enhancements in order for spatial scalability to be fully supported. I’m adding him to the Cc, as well as my co-authors on the VP9 payload.

Sergio, if you want to send a pull request to contribute text for the VP9 spec (which would be welcome), it’s in Github at https://github.com/juberti/draughts .
Currently the VP9 LID mapping is described on the frame marking draft, and not in the VP9 draft. We could move the full description from one text to another, but at least for me, it was easier to provide a small patch for current draft to address my comments:

https://github.com/murillo128/draft-berger-avtext-framemarking/commit/35cdb229108251f12bc1cdf77909df83a51d3994

I have not found the .xml in any github repo, so just downloaded the latest draft and applied the changes on top of it.

Please, let me know what are your views about it.

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 5 Jul 2017 09:52:14 -0700

Jonathan said:

"the “don’t use it with weird structures” requirement will make that
sufficient,"

[BA] There are structures where the B bit will not provide enough
information to indicate that upscaling is possible (e.g. B bit 0 would
indicate that the frame doesn't depend solely on the base layer, but P and
U bits would indicate upscaling is possible).

A "weird structures requirement" is somewhat unnatural. If the structures
are not illegal in the bitstream specification, encoders can generate them,
and decoders will be required to be able to decode them.

This can generate a mismatch between what SFUs are built to handle and what
endpoints do.  If the payload is encrypted, the SFU won't necessarily be
able to detect a "weird" structure and it can't act based on the P and U
bits, only the B bit. Even if the payload is unencrypted, the SFU may wish
to function based on the frame marking header alone, so as to avoid parsing
the payload for the P and U bits.

On Wed, Jul 5, 2017 at 8:56 AM, Jonathan Lennox <[email-masked]> wrote:

> Sergio — the updated VP9 payload draft I submitted Monday includes a
> section describing its usage with frame marking.
>
> Let me know what you think?  It changes the S and T fields to reflect the
> latest VP9 and frame marking drafts, but it does not include the P or U
> bits.
>
> I think the intention is that frame marking’s B bit will provide a
> coarser-grain version of what P and U do (and the “don’t use it with weird
> structures” requirement will make that sufficient, though that may need to
> be made more specific).  What do you think?
>
> On Jun 28, 2017, at 10:23 AM, Sergio Garcia Murillo <
> [email-masked]> wrote:
>
> Yes, at least in VP9 SVC you can define how many temporal and how many
> spatial layers do you want simultaneously in the encoding. Note also, that
> VP9 supports quality layers as spatial layers without any resolution
> change, so the term "spatial layer" is used to represent both spatial and
> quality layers.
> BR
> Sergio
> Sergio said:  "So, shouldn't the LID reference to the spatial layer ID
> only and omit the quality layer id completely? Also, the spatial layer id
> is 3 bits on that draft."
>
> [BA] Question:  In VP9 (or AV1, for that matter) is it possible to use
> both spatial and quality scalability in the same encoding?
>
> On Wed, Jun 28, 2017 at 5:35 AM, Sergio Garcia Murillo <
> [email-masked]> wrote:
>
>> Hi all again,
>>
>> Did you had a chance to review the changes to add the VP9 SVC spatial
>> layer information into the LID of the frame marking extension? (at
>> https://github.com/murillo128/draft-berger-avtext-framemarki
>> ng/commit/35cdb229108251f12bc1cdf77909df83a51d3994)
>>
>> Also, FYI, I have submitted a CL to libwebrtc to support framemarking
>> extension. Currently it supports VP8, H264 and non-SVC VP9, but I would
>> love to tackle VP9 SVC as soon as we get an agreement on the changes to the
>> LID:
>>
>>>> One of the goals of Frame Marking is that an SFU can be agnostic to the
>>>> details of the layering, and just treat layer IDs as opaque layer
>>>> identifiers.  Putting non-layer bits into the LID field would break this
>>>> goal.
>>>>
>>>> At the IETF meeting in Chicago, both Bernard Aboba and I thought that
>>>> frame marking might need enhancements in order for spatial scalability to
>>>> be fully supported. I’m adding him to the Cc, as well as my co-authors on
>>>> the VP9 payload.
>>>>
>>>> Sergio, if you want to send a pull request to contribute text for the
>>>> VP9 spec (which would be welcome), it’s in Github at
>>>> https://github.com/juberti/draughts .
>>>>
>>> Currently the VP9 LID mapping is described on the frame marking draft,
>>> and not in the VP9 draft. We could move the full description from one text
>>> to another, but at least for me, it was easier to provide a small patch for
>>> current draft to address my comments:
>>>
>>> https://github.com/murillo128/draft-berger-avtext-framemarki
>>> ng/commit/35cdb229108251f12bc1cdf77909df83a51d3994
>>>
>>> I have not found the .xml in any github repo, so just downloaded the
>>> latest draft and applied the changes on top of it.
>>>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Wed, 5 Jul 2017 19:30:20 +0200

On 05/07/2017 17:56, Jonathan Lennox wrote:
> Let me know what you think?  It changes the S and T fields to reflect 
> the latest VP9 and frame marking drafts, but it does not include the P 
> or U bits.
>
> coarser-grain version of what P and U do (and the “don’t use it with 
> weird structures” requirement will make that sufficient, though that 
> may need to be made more specific).  What do you think?

An SFU requires both VP9  P and U bits in order to be able to do up and 
down switching. So the question is they can be mapped 1-to-1 to the 
frame marking I and B bits. If so, just adding that to the draft that 
the I and B bits on the frame marking extension must be filled with the 
values of the vp9 P and U bits ( I=!P , B=U), would work.

By the way, I liked more the "layer frame" and "frame" vs "frame" 
"picture" (but it is just a personal preference), but I am not sure if 
there haven't been any typos on the renaming, I will have to review it 
in deep because there are some things that seems wrong at first glance 
(but I am probably wrong). Will send a new email if I find anything.

Best regards
Sergio

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 5 Jul 2017 14:19:43 -0700

Sergio said:

"So the question is they can be mapped 1-to-1 to the frame marking I and B
bits. If so, just adding that to the draft that the I and B bits on the
frame marking extension must be filled with the values of the vp9 P and U
bits ( I=!P , B=U), would work."

[BA] It's not obvious to me that the P or U bits can be inferred from
existing framemarking bits including the I and B bits.  Consider the
following dependency diagrams (done in the style of the examples in
https://tools.ietf.org/html/draft-ietf-avtext-lrr ):

Example 1

        ... <--  S2  <--  S2       S2  <--  S2  <-- ...
                  |        |        |        |

                                 \/              \/               \/
           \/

        ... <--  S1  <--  S1       S1  <--  S1  <-- ...
                  |        |        |        |
                 \/       \/       \/       \/
        ... <--  S0  <--  S0  <--  S0  <--  S0  <-- ...

                  1        2        3        4

In this example, it looks to me like all the S0 frames have the B bit
set to 1, but none of the S0, S1 or S2 frames have the I bit set to 1.

Frames 3 S1 and S2 have the B bit set to 1, all other S1 and S2 frames
have the B bit set to zero.

It also looks to me like only frames 3 S1 and S2 have the P bit set to
zero, all other S0, S1 and S2 frames have it set to 1.

So there is no clear relationship between the I and P bits.

Example 2

        ... <--  S2  <--  S2  <--  S2  <--  S2  <-- ...
                  |        |        |        |

                                 \/              \/               \/
           \/

        ... <--  S1  <--  S1  <--  S1  <--  S1  <-- ...
                  |        |        |        |
                 \/       \/       \/       \/
        ... <--  S0  <--  S0       S0  <--  S0  <-- ...

                  B        B       BIP      B

                  1        2        3        4

In this example, it looks to me like all the S0 frames have the B bit
set to 1, but only S0 frame 3 has the I bit set to 1.

None of the S1 or S2 frames have the B or I bits set to 1.  It also
looks to me like none of the frames have the P bit set to 0 or the U

bits set to 1, except frame 3 S0.  So in this example, the I and P
bits are the inverse of each other, but it's not clear how the

U bit can be inferred from other bits.

Here are the definitions of the framemarking I, D and B bits:

   o  I: Independent Frame (1 bit) - MUST be 1 for frames that can be
      decoded independent of prior frames, e.g. intra-frame, VPX
      keyframe, H.264 IDR [RFC6184
<https://tools.ietf.org/html/rfc6184>], H.265 IDR/CRA/BLA/RAP [RFC7798
<https://tools.ietf.org/html/rfc7798>];
      otherwise MUST be 0.
   o  D: Discardable Frame (1 bit) - MUST be 1 for frames that can be
      discarded, and still provide a decodable media stream; otherwise
      MUST be 0.
   o  B: Base Layer Sync (1 bit) - MUST be 1 if this frame only depends
      on the base layer; otherwise MUST be 0.  If no scalability is
      used, this MUST be 0.

Here are the P and U bit definitions:

   P: Inter-picture predicted layer frame.  When set to zero, the layer
      frame does not utilize inter-picture prediction.  In this case,
      up-switching to current spatial layer's frame is possible from
      directly lower spatial layer frame.  P SHOULD also be set to zero
      when encoding a layer synchronization frame in response to an LRR
      [I-D.ietf-avtext-lrr
<https://tools.ietf.org/html/draft-ietf-payload-vp9-03#ref-I-D.ietf-avtext-lrr>]
message (see Section 5.4
<https://tools.ietf.org/html/draft-ietf-payload-vp9-03#section-5.4>).
When P is set to
      zero, the T bit (described below) MUST also be set to 0 (if
      present).

   U: Switching up point.  If this bit is set to 1 for the current
      frame with temporal layer ID equal to T, then "switch up" to a
      higher frame rate is possible as subsequent higher temporal
      layer frames will not depend on any frame before the current
      frame (in coding time) with temporal layer ID greater than T.

On Wed, Jul 5, 2017 at 10:30 AM, Sergio Garcia Murillo <
[email-masked]> wrote:

> down switching. So the question is they can be mapped 1-to-1 to the frame
> marking I and B bits. If so, just adding that to the draft that the I and B
> bits on the frame marking extension must be filled with the values of the
> vp9 P and U bits ( I=!P , B=U), would work.
>
> there haven't been any typos on the renaming, I will have to review it in
> deep because there are some things that seems wrong at first glance (but I
> am probably wrong). Will send a new email if I find anything.
>
> Best regards
> Sergio
>
> _______________________________________________
> avtext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avtext
>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Wed, 5 Jul 2017 23:41:16 +0200

On 05/07/2017 23:19, Bernard Aboba wrote:
> Sergio said:
>
> "So the question is they can be mapped 1-to-1 to the frame marking I 
> and B bits. If so, just adding that to the draft that the I and B bits 
> on the frame marking extension must be filled with the values of the 
> vp9 P and U bits ( I=!P , B=U), would work."
>
> existing framemarking bits including the I and B bits.

I agree, it is far from obvious.. :)

The point is that there is no easy way of calculating the B bit on frame 
marking for VP9 (you would need to process the SS to calculate it as you 
have done in the examples, but that info is not provided by the encoder 
neither present on the desc header), so the only close-enough info would 
be the U bit. If we "redefine" it's meaning for VP9 it could work.

Regarding the P bit, in prev draft the information was per layer-frame, 
now with the renaming, it is per picture. Not sure if it is a typo, or 
it has changed it's meaning, but if so, that would be exactly the 
inverse of the I bit.

Best regards
Sergio

---

From: Jonathan Lennox <[email-masked]>
Date: Wed, 5 Jul 2017 22:17:38 +0000

Adding payload@ietf, since details of the VP9 spec itself (as opposed to how it’s used with frame marking) should be discussed there.  Please restrict followups as appropriate.

> On Jul 5, 2017, at 5:41 PM, Sergio Garcia Murillo <[email-masked]> wrote:
> 
> Regarding the P bit, in prev draft the information was per layer-frame, now with the renaming, it is per picture. Not sure if it is a typo, or it has changed it's meaning, but if so, that would be exactly the inverse of the I bit.

This was a typo, yes. Thanks for catching it!  I’ll fix it in the next version. (Notice that there’s a lot of other text around that definition that doesn’t make any sense if P is per-picture.)

On the reasons for the terminology change: “super frame” already means something (different) in VP9, so reusing the term was problematic.  Additionally, what we now call “frame” is exactly what the VP9 codec standard means by “frame”, i.e. a single encoded image.