Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

From: Jonatan Samuelsson <[email-masked]>
Date: Wed, 14 May 2014 08:57:42 +0000

Dear Ye-Kui, editors, all,
Thanks for improving the draft and making the new version available. I read through the new section 8 in the latest draft, in which more feedback features have been added (which I think is very good). I have some minor comments and then some suggestions for improving the RPSI signalling. Compared to earlier standards, HEVC contains a new very useful SEI message that can be used to check integrity and correctness of decoded pictures and thereby detect errors that are not necessarily due to packet losses: Decoded  picture hash SEI (e.g. MD5 calculated over a decoded picture) . I think it would be good to exploit that functionality also in the RTP payload format for HEVC in order to make implementations even more robust.
First some minor comments:
In 8.1:
A. Remove one instance of 'the loss of' in the sentence: ' As specified in RFC 4585 section 6.3.1, the reception of a picture loss indication by a media sender indicates the loss of "the loss of an undefined amount of coded video data belonging to one or more pictures.".'
B. The 'BE' in 'SHOULD BE' should not be using CAPITAL LETTERS.
In 8.2:
C. Regarding 'the loss of a number of Coded Tree Blocks (CTBs) in CTB raster scan order of a picture.'. When tiles are used, the loss of a slice does typically not result in lost CTBs in raster scan order of the picture. The text does not describe how to calculate the "Number" field for this case. Wouldn't it be better to use 'tile scan' in order to correctly identify what part of the picture was lost?
D. Regarding 'the subfield "First" MUST be set to the CTB address of the first lost CTB'. Wouldn't it be better to require "First" to be set to a number that is lower than or equal to the CTB address of the first lost CTB, since it might be difficult to deduce the exact CTB address of the first lost CTB in the case of loss of  dependent slices and/or fragmentation units?
E. Regarding 'The subfield "PictureID" MUST  be  set  to  the  6  least significant  bits  of  a  binary representation  of  the  value  of slice_pic_order_cnt_lsb  of  the picture for which the lost CTBs are indicated.'. Isn't it better to use the 6 least significant bits of PicOrderCntVal? Using the 6 least significant bits of  slice_pic_order_cnt_lsb seems a bit confusing (and less robust) for the case when slice_pic_order_cnt_lsb consists of  fewer than 6 bits.
F. Regarding the sentence: 'In many cases, error tracking is required to identify the corrupted region in the receiver's state (reference pictures) because of error import in uncorrupted regions of the picture through motion compensation, and reference picture selection can also be used to "clean up" the corrupted picture, which is usually more efficient and less likely to generate congestion than sending intra information.' For improved readability I suggest splitting it in two sentences: 'In many cases, error tracking is required to identify the corrupted region in the receiver's state (reference pictures) because of error import in uncorrupted regions of the picture through motion compensation. Reference picture selection can also be used to "clean up" the corrupted picture, which is usually more efficient and less likely to generate congestion than sending intra information.'
Regarding RPSI (section 8.3) I would like to propose the following changes:
G. Add the possibility to include more than one reference picture in the RPSI message so that the encoder can select to use one or more of the pictures included in RPSI message. Using multiple pictures for reference is a key feature in HEVC which substantially improves coding efficiency. In the case where multiple earlier pictures are available for reference it is beneficial to let the encoder decide which one(s) to use for reference instead of relying on the decoder to make that selection.
H. Using the 2 currently unused bits in the 'Native RPSI bit string defined per codec' for adding the possibility of including information regarding the MD5 in the RPSI message as follows:
      0:            No information regarding the MD5 available
      1:            The MD5 of the reference picture is verified to be correct
      2:            The MD5 of the reference picture is incorrect (i.e. do NOT use for reference)
      3:            The MD5 is included in the RPSI message (directly following the PicOrderCntVal)
The last option can be used when the encoder does not include MD5s in-band but wants to check the MD5 before using a selected picture for reference. Typically this will be more bit-efficient than including decoded picture hash SEI messages in-band.
In my opinion it would be very useful to be able to include information regarding the correctness of the decoded pictures, e.g. to detect incompatibility issues and bit errors. I will send a suggestion for the text changes needed to enable this, hopefully later today.
Best Regards Jonatan
-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 30 april 2014 20:29
To: [email-masked]
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear All,

Compare to -02, we have addressed most of the numerous comments and tickets, from Magnus, Jonathan, Jonatan, Bernard, ..., as well as the following changes:
- Addition of sprop-sei, as suggested by Stephan on Apr. 23 to the reflector, agreed by most of the authors, no objection from anyone.
- Addition of an informative, brief description of the display orientation SEI message, which is among one of those new SEI messages that have some systems usage.
- Addition of a definition of NAL-unit-like structure. The term was used but not defined earlier.
- Addition of descriptions for use of the PLI and SLI messages (specified in RFC 4585) and the FIR message (specified in RFC 5104) with HEVC, in addition to the RFC 4585 RPSI message for which the use description was already included, as agreed during the presentation at the previous IETF meeting

Remaining open issues:
- On profile-compatibility-indicator and interop-constraints (raised by Magnus, being discussed among Magnus, myself and Yago, no conclusion yet)
- On recv-sub-layer-id and using of sprop-vps for session negotiation (raised by Magnus, still being discussed among the authors and Magnus)
- On source-specific sprop parameter sets (raised by Magnus, still being discussed among the authors and Jonathan for exact text changes)
- On using a different parameter category for level-id (raised by Magnus, discussed between Magnus and myself, no conclusion yet)
- I just noticed that the definition of "packet stream" itself needs to be changed back to "RTP stream"

We will try to address the above open issues as soon as possible. At the same time, further review and comments are welcome.

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, April 30, 2014 10:34 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Audio/Video Transport Payloads Working Group of the IETF.

        Title           : RTP Payload Format for High Efficiency Video Coding
        Authors         : Ye-Kui Wang
                          Yago Sanchez
                          Thomas Schierl
                          Stephan Wenger
                          Miska M. Hannuksela
        Filename        : draft-ietf-payload-rtp-h265-03.txt
        Pages           : 92
        Date            : 2014-04-30

Abstract:
   This memo describes an RTP payload format for the video coding
   standard ITU-T Recommendation H.265 and ISO/IEC International
   Standard 23008-2, both also known as High Efficiency Video Coding
   (HEVC) [HEVC] and developed by the Joint Collaborative Team on Video
   Coding (JCT-VC).  The RTP payload format allows for packetization of
   one or more Network Abstraction Layer (NAL) units in each RTP packet
   payload, as well as fragmentation of a NAL unit into multiple RTP
   packets.  Furthermore, it supports transmission of an HEVC bitstream
   over a single as well as multiple RTP streams.  The payload format
   has wide applicability in videoconferencing, Internet video
   streaming, and high bit-rate entertainment-quality video, among
   others.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-h265/

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-payload-rtp-h265-03

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-payload-rtp-h265-03

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
payload mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/payload

_______________________________________________
payload mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/payload

---

From: Jonatan Samuelsson <[email-masked]>
Date: Thu, 15 May 2014 20:25:18 +0000

Dear all,

Magnus and I have written text for enabling the use of decoded picture hash (e.g. MD5) information in RPSI feedback messages. We have made sure that the syntax to a large extent is compatible with earlier versions of the draft in case anyone has started to implement HEVC RPSI already (since the MT value equal to 0 corresponds to that no information regarding the decoded picture hash is available in the RPSI message).

The suggestion includes:

-          The possibility to include more than one reference picture in an RPSI feedback message

-          The possibility to indicate information about decoded picture hash (e.g. MD5) for each picture in an RPSI feedback message

-          A parameter, include-dph, for indicating/requesting that decoded picture hash SEI will be sent with the video data

-          A parameter, rpsi-dph, for indicating/requesting that decoded picture hash will be included in RPSI feedback messages

Here is our suggested text:

8.3 Use of HEVC with the RPSI Feedback Message

Feedback based reference picture selection has been shown as a powerful tool to stop temporal error propagation for improved error resilience [Girod99][Wang05].  In one approach, the decoder side tracks errors in the decoded pictures and informs to the encoder side that one or more particular pictures that has been decoded relatively earlier is correct and still present in the decoded picture buffer and requests the encoder to use one or more of those correct pictures for reference when encoding the next picture, so to stop further temporal error propagation.  For this approach, the decoder side should use the RPSI feedback message [RFC4585].

Encoders can encode some long-term reference pictures as specified in H.264 or HEVC for purposes described in the previous paragraph without the need of a huge decoded picture buffer.  As shown in [Wang05], with a flexible reference picture management scheme as in H.264 and HEVC, even a decoded picture buffer size of two would work for the approach described in the previous paragraph. The hash values, as defined in section D.3.19 in [HEVC] can be used to verify the correctness of decoded pictures.

Informative note: The use of hash values calculated over the decoded sample values enables detection of problems that are not necessarily due to packet losses. Examples of such errors are; bit-errors in the transport, memory corruption in decoder or encoder and encoder-decoder incompatibility problems (i.e. due to non-conforming encoder or decoder).

For HEVC, the field "Native RPSI bit string defined per codec" in the RPSI message consists of one or more entries that follows the format depicted in Figure X:
   0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | MT| RefLayer  |                 RefPicId                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    RefPicId   |                                               |
   +-+-+-+-+-+-+-+-+                                               +
   |                                                               |
   +             Decoded Picture Hash (conditional)                +
   |                                                               |
   +                                                               +
   |                                                               |
   +                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure X: The HEVC "Native RPSI bit string defined per codec"

The semantics of the fields are as follows:

MT: 2 bits
     Indicates the hash information as follows:
           0:   No hash information regarding the correctness of the reference picture is available.
           1:   The Decoded Picture Hash SEI has been used to verify the reference picture to be correct.
           2:   The Decoded Picture Hash SEI has been used to verify the reference picture to be incorrect.
           3:   The Decoded Picture Hash of the reference picture is included in the RPSI message.
When MT equals 0, 1 or 3, the reference picture identified by the current entry is requested to be used for reference when encoding the next picture. For 3 with the exception, if the encoder finds the provided hash of the reference picture to not match the encoder's hash value, in which case it MUST NOT use the reference picture.

Informative note: When an RPSI message contains more than one entry with MT equal to 0, 1 or 3 the encoder may select one or more of the identified pictures to be used for reference. The selection of which picture(s) to use for reference is out of scope of this specification but may for example be based on maximizing compression efficiency.

When MT equals 2 the reference picture identified by the current entry MUST NOT be used for reference for the next picture or any picture that follows the next picture.

RefLayer: 6 bits
Represents the 6 bits of nuh_layer_id of the reference picture, as defined in [HEVC].

RefPicId: 32 bits
Represents the value of the PicOrderCntVal (in network byte order) of the reference picture, as defined in [HEVC].

Decoded Picture Hash: Variable number of bytes
Present only if MT equals 3. Represent the Decoded Picture Hash SEI payload (in network byte order), see D.2.19 of [HEVC], of the decoded picture. The Decoded Picture Hash payload starts with a one byte type field, which gives the amount of hash data (2, 4 or 16 bytes for the defined hashes).

The use of the RPSI feedback message as positive acknowledgement with HEVC is deprecated.  In other words, the RPSI feedback message MUST only be used as a reference picture selection request, such that it can also be used in multicast.

Section 7.1
New parameters:

include-dph:
This parameter is used to indicate capability and preference to utilize or include Decoded Picture Hash (DPH) SEI messages (See Section D.3.19 of [HEVC]) in the encoded video sequence. The value is a comma separated list of hash types that is supported or requested to be used, each hash type provided as an unsigned integer value (0-255), with the hash types listed from most preferred to the least preferred. Example: "include-dph=0,2", which indicates the capability for MD5 (most preferred) and Checksum (less preferred). If the parameter is not included or the value contains no hash types, then no capability to utilize DPH SEI messages is assumed. Note that DPH SEI messages MAY still be included in the encoded video sequence even when there is no declaration of capability to use them.

rpsi-dph:
This parameter is used to indicate capability and preference for which types of Decoded Picture Hash to include or use with the RTCP Reference Picture Selection Indication (RPSI) message as defined in Section 8.3. The value is a comma separated list of hash types that is supported or requested to be used, each hash type provided as an unsigned integer value (0-255), with the hash types listed from most preferred to the least preferred. If the parameter is not included or the value contains no hash types, then no capability to utilize DPH in RPSI is assumed. In this case, RTCP RPSI entries using MT=3 SHALL NOT be sent, MT=0,1, and 2 MAY still be sent. This parameter has meaning only if support for RTCP RPSI messages for the HEVC RTP payload type is indicated.

Section 7.2.1:
Add above parameters in list of Optional parameters

Section 7.2.2:
*    The capability parameter include-dph MAY be used to declare the capability to utilize Decoded Picture Hash SEI messages and which types of hashes in any RTP streams received by the offerer or answerer. The parameter MAY be included in offerer or answerer's media descriptions that are a=sendonly to indicate a capability to provide DPH SEIs.
*    The capability parameter rpsi-dph MAY be used to declare an offerer or answerer's capability to receive RPSI messages which include Decoded Picture Hash and which types of hashes. This parameter SHOULD only be provided when RTCP RPSI is indicated. Note that this parameter is media sender property, not a receiver property. Thus, for media descriptions with sendonly or sendrecv, the declaring entity (offerer or answerer) has the capability to receive RPSI with the hashes. For a media description that is a=recvonly the inclusion of the parameter indicates the capability of media receiver to provide hashes in the RPSI it may send.

Section 7.2.3:
Add include-dph and rpsi-dph to the list "Declaring actual configuration or bitstream properties:"

Best Regards Jonatan and Magnus

From: payload [mailto:[email-masked]] On Behalf Of Jonatan Samuelsson
Sent: den 14 maj 2014 10:58
To: Wang, Ye-Kui; [email-masked]
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 30 april 2014 20:29
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear All,

Remaining open issues:

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, April 30, 2014 10:34 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Abstract:

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 16 May 2014 23:09:37 +0000

Hi Jonatan,

Thanks for reviewing the new version. Please see my replies inline below, including some comments on the two newly proposed features related to RPSI. For these two features, let's first discuss whether to have them in this payload format document before going into the details in text proposed in another email.

BR, YK

From: Jonatan Samuelsson [mailto:[email-masked]]
Sent: Wednesday, May 14, 2014 1:58 AM
To: Wang, Ye-Kui; [email-masked]
Subject: RE: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

[YK]: Thanks! To be done in the next version.
B. The 'BE' in 'SHOULD BE' should not be using CAPITAL LETTERS.
[YK]: Thanks! To be done in the next version.

In 8.2:

[YK]: The intended use of the SLI message as specified is to indicate the loss of CTBs regardless of whether and how slices are in use and whether and how tiles are in use. The semantics of the "Number" field is clear to me too - why it is not clear? Note that the semantics of slice_segment_address in the HEVC spec itself is also in CTB raster scan order, regardless of whether and how tiles are in use. Personally I'd prefer to keep the semantics of SLI to be generic, and also aligned with the semantics of slice_segment_address in the HEVC spec.
D. Regarding 'the subfield "First" MUST be set to the CTB address of the first lost CTB'. Wouldn't it be better to require "First" to be set to a number that is lower than or equal to the CTB address of the first lost CTB, since it might be difficult to deduce the exact CTB address of the first lost CTB in the case of loss of  dependent slices and/or fragmentation units?

[YK]: To me this change would make the semantics a lot less clear and the message less useful. Also, each dependent slice segment has slice_segment_address signalled too, while FUs are supposed to be assembled before sending to the decoder - thus from the decoder point of view FUs do not need to be considered.
E. Regarding 'The subfield "PictureID" MUST  be  set  to  the  6  least significant  bits  of  a  binary representation  of  the  value  of slice_pic_order_cnt_lsb  of  the picture for which the lost CTBs are indicated.'. Isn't it better to use the 6 least significant bits of PicOrderCntVal? Using the 6 least significant bits of  slice_pic_order_cnt_lsb seems a bit confusing (and less robust) for the case when slice_pic_order_cnt_lsb consists of  fewer than 6 bits.
[YK]: Good point. To be done in the next version unless the discussion leads into a different conclusion.

F. Regarding the sentence: 'In many cases, error tracking is required to identify the corrupted region in the receiver's state (reference pictures) because of error import in uncorrupted regions of the picture through motion compensation, and reference picture selection can also be used to "clean up" the corrupted picture, which is usually more efficient and less likely to generate congestion than sending intra information.' For improved readability I suggest splitting it in two sentences: 'In many cases, error tracking is required to identify the corrupted region in the receiver's state (reference pictures) because of error import in uncorrupted regions of the picture through motion compensation. Reference picture selection can also be used to "clean up" the corrupted picture, which is usually more efficient and less likely to generate congestion than sending intra information.'
[YK]: To be done in the next version.

Regarding RPSI (section 8.3) I would like to propose the following changes:
G. Add the possibility to include more than one reference picture in the RPSI message so that the encoder can select to use one or more of the pictures included in RPSI message. Using multiple pictures for reference is a key feature in HEVC which substantially improves coding efficiency. In the case where multiple earlier pictures are available for reference it is beneficial to let the encoder decide which one(s) to use for reference instead of relying on the decoder to make that selection.
[YK]: A few points to discuss. G.1: Is it backward compatible to RFC 4585 to use one RPSI message to convey more than one reference picture? Would it be better to define a new message for this purpose if it is deemed useful? G.2: The concept applies to any video codec that supports multiple reference pictures, is not specific to HEVC, thus would it be better if this is discussed as a generic mechanism, similarly as the SPLI message included in draft-ietf-payload-rtp-h265-01 but dropped later for the very reason? G.3: On the usefulness of this functionality seems to be unclear to me either. If the last correct picture before an error occurs is known, it is also known that the reference pictures of the last correct pictures are correct, thus the encoder can anyway choose to use one or more of these pictures, when available, for encoding the next picture. Thus, it might be sufficient just to slightly change the semantics of the RPLI message, to say that the encoder can choose to use one or more of the identified picture and its reference pictures for encoding the next picture.

[YK]: Firstly, in my understanding, the decoder picture hash SEI message is very useful for debugging in development of encoder and decoder implementations, as used by the JCT-VC for the development of the HEVC reference software, and it is intended for use with that purpose. Using of it for error resilience in communications is an overkill to me. For example, you would need 48 bytes to signal the MD5 hash for each picture. Secondly, it is architecturally incorrect to couple this with the RPSI feedback message, as the picture identified is supposed to be correctly decoded by the decoder and the purpose of the message is to request the encoder to use that picture for encoding of the next picture, thus why would you even indicate a picture that is incorrect using an RPSI feedback message?

Best Regards Jonatan
-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 30 april 2014 20:29
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear All,

Remaining open issues:

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, April 30, 2014 10:34 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Abstract:

---

From: Yago Sanchez <[email-masked]>
Date: Sat, 17 May 2014 12:46:45 +0200

Hi Jonatan, 

Similar to Ye-Kui’s comments, on the inclusion of MD5 in the RPSI, it is not clear to me why this could be useful. When talking about “incompatibility” about encoder and decoder this would mean that one of them is not properly working, i.e. it does not fully conform to the standard. In such a case I personally do not think that this is an issue needed to be solved with the RPSI but at the development phase of the encoder/decoder. I would assume here that both encoder and decoder are conforming to the standard.

For other cases, I cannot think of any case of a picture being “correctly” decoded, while being a mismatch between encoder and decoder. Is there any case where this happens that I am missing?

Best regards,
Yago

On 17 May 2014, at 01:09, Wang, Ye-Kui <[email-masked]> wrote:

> Hi Jonatan,
>  
> Thanks for reviewing the new version. Please see my replies inline below, including some comments on the two newly proposed features related to RPSI. For these two features, let’s first discuss whether to have them in this payload format document before going into the details in text proposed in another email.
>  
> BR, YK
>  
> [YK]: The intended use of the SLI message as specified is to indicate the loss of CTBs regardless of whether and how slices are in use and whether and how tiles are in use. The semantics of the “Number” field is clear to me too – why it is not clear? Note that the semantics of slice_segment_address in the HEVC spec itself is also in CTB raster scan order, regardless of whether and how tiles are in use. Personally I’d prefer to keep the semantics of SLI to be generic, and also aligned with the semantics of slice_segment_address in the HEVC spec.
> [YK]: To me this change would make the semantics a lot less clear and the message less useful. Also, each dependent slice segment has slice_segment_address signalled too, while FUs are supposed to be assembled before sending to the decoder – thus from the decoder point of view FUs do not need to be considered.
> E. Regarding 'The subfield "PictureID" MUST  be  set  to  the  6  least significant  bits  of  a  binary representation  of  the  value  of slice_pic_order_cnt_lsb  of  the picture for which the lost CTBs are indicated.'. Isn't it better to use the 6 least significant bits of PicOrderCntVal? Using the 6 least significant bits of  slice_pic_order_cnt_lsb seems a bit confusing (and less robust) for the case when slice_pic_order_cnt_lsbconsists of  fewer than 6 bits.
> From: payload [mailto:[email-masked]] On Behalf Of [email-masked]
> To: [email-masked]
> Cc: [email-masked]
> [email-masked]
> [email-masked]
> [email-masked]

---

From: Jonatan Samuelsson <[email-masked]>
Date: Mon, 19 May 2014 13:28:44 +0000

Dear Ye-Kui, Yago, all,

Let me start with some further motivation for why I think it would be useful to include MD5 information in RPSI. I see two categories of "errors" that can cause encoder and decoder MD5 mismatch:

1.       External causes. It is possible that a bit-error in a bitstream is not detected during decoding, causing a decoded picture to have different values compared to the reconstructed picture at the encoder. It is also possible that decoded pictures or reconstructed pictures are stored on a media that is not 100% reliable (e.g. due to malfunctioning hardware). Having a single value wrong in a reference picture can quite fast propagate to severe visual artifacts due to temporal prediction.

2.       Implementation causes. Each HEVC encoder and decoder implementation (both hardware and software) is an interpretation of the more than 300 pages long HEVC specification and there is always the risk that there are some parts of the specification interpreted differently. The reference encoder and decoder (HM) can perform guidance here but HM is far from "complete" in the sense that it can test all encoder (bitstream) features fully or correctly report all illegal bitstreams to be non-conforming. There is also the risk that an implementation contains bugs that appear only for very rare cases.

Our motivation for including MD5 in RPSI is primarily based on the second category (although the first category should not be ignored). It is true that you would ideally want all deployed encoders and decoders to be fully standards compliant with no exceptions. However, history has taught us that this is unfortunately not the case. For h.264, even in the reference software JM, bugs have been found several years after h.264 was standardized (example: https://ipbt.hhi.fraunhofer.de/mantis/view.php?id=193). And when x264 started using weighted prediction several decoders silently accepted those bitstreams while displaying erroneous video (http://x264dev.multimedia.cx/archives/212#more-212). I would not expect HEVC with all its different possible combinations of block sizes and new coding tools to be easier to implement correct compared to h.264.

We believe it would be very useful to provide better tools for discovering incompatibility issues and be able to automatically trigger events based on the detection of MD5 mismatch. It could for example include:

-          Taking immediate action to solve the problem such as sending an IDR picture

-          Change the configuration of a session with the hope of finding a configuration which does not cause encoder-decoder problems (e.g. by using a more basic set of tools or by switching profie/level)

-          Gather statistics for which clients does not work with which clients and under which circumstances

See further comments inline.

Best Regards Jonatan

From: Yago Sanchez [mailto:[email-masked]]
Sent: den 17 maj 2014 12:47
To: Jonatan Samuelsson
Cc: [email-masked]; Wang, Ye-Kui
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Hi Jonatan,

Similar to Ye-Kui's comments, on the inclusion of MD5 in the RPSI, it is not clear to me why this could be useful. When talking about "incompatibility" about encoder and decoder this would mean that one of them is not properly working, i.e. it does not fully conform to the standard. In such a case I personally do not think that this is an issue needed to be solved with the RPSI but at the development phase of the encoder/decoder. I would assume here that both encoder and decoder are conforming to the standard.

For other cases, I cannot think of any case of a picture being "correctly" decoded, while being a mismatch between encoder and decoder. Is there any case where this happens that I am missing?

Best regards,
Yago

On 17 May 2014, at 01:09, Wang, Ye-Kui <[email-masked]<mailto:[email-masked]>> wrote:

Hi Jonatan,

BR, YK

From: Jonatan Samuelsson [mailto:[email-masked]]
Sent: Wednesday, May 14, 2014 1:58 AM
To: Wang, Ye-Kui; [email-masked]<mailto:[email-masked]>
Subject: RE: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

In 8.2:

[YK]: The intended use of the SLI message as specified is to indicate the loss of CTBs regardless of whether and how slices are in use and whether and how tiles are in use. The semantics of the "Number" field is clear to me too - why it is not clear? Note that the semantics of slice_segment_address in the HEVC spec itself is also in CTB raster scan order, regardless of whether and how tiles are in use. Personally I'd prefer to keep the semantics of SLI to be generic, and also aligned with the semantics of slice_segment_address in the HEVC spec.
[JS] I guess I'm  missing something here. Let's look at a very simple example of a picture of 4x3 CTBs with two tiles, one to the right and one to the left, each signaled in a separate slice.
The CTB addresses in CTB raster scan order of the picture is:
-------------
| 0| 1| 2| 3|
| 4| 5| 6| 7|
| 8| 9|10|11|
-------------
With two tiles layed out as follows:
-------------
| A| A| B| B|
| A| A| B| B|
| A| A| B| B|
-------------
the CTB address in tile scan is:
-------------
| 0| 1| 6| 7|
| 2| 3| 8| 9|
| 4| 5|10|11|
-------------
Now, if the second tile (second slice) is lost. I should obviously set the value of "First" to 2 (since that is the CTB address in CTB raster scan order of the picture), but what value should I set for "Number"? I would like to set it to 6 since that is the number of CTBs that are lost. But if I count in CTB raster scan order of the picture I get to 10 before I reach the last CTB that was lost. This means some CTBs needs to be indicated as lost even though they were not. Or do I misinterpret the following sentence:
The subfield "Number" MUST be set to the number of consecutive lost CTBs, again in CTB raster scan order of a picture.
D. Regarding 'the subfield "First" MUST be set to the CTB address of the first lost CTB'. Wouldn't it be better to require "First" to be set to a number that is lower than or equal to the CTB address of the first lost CTB, since it might be difficult to deduce the exact CTB address of the first lost CTB in the case of loss of  dependent slices and/or fragmentation units?

[YK]: To me this change would make the semantics a lot less clear and the message less useful. Also, each dependent slice segment has slice_segment_address signalled too, while FUs are supposed to be assembled before sending to the decoder - thus from the decoder point of view FUs do not need to be considered.
[JS] Sending incomplete FUs to the decoder can significantly improve error concealment algorithms. However, this is just a minor comment from my side. It is anyway up to the receiver to decide how much data is reported to be lost regardless of how much it actually received.
E. Regarding 'The subfield "PictureID" MUST  be  set  to  the  6  least significant  bits  of  a  binary representation  of  the  value  of slice_pic_order_cnt_lsb  of  the picture for which the lost CTBs are indicated.'. Isn't it better to use the 6 least significant bits of PicOrderCntVal? Using the 6 least significant bits of  slice_pic_order_cnt_lsb seems a bit confusing (and less robust) for the case when slice_pic_order_cnt_lsbconsists of  fewer than 6 bits.
[YK]: Good point. To be done in the next version unless the discussion leads into a different conclusion.

Regarding RPSI (section 8.3) I would like to propose the following changes:
G. Add the possibility to include more than one reference picture in the RPSI message so that the encoder can select to use one or more of the pictures included in RPSI message. Using multiple pictures for reference is a key feature in HEVC which substantially improves coding efficiency. In the case where multiple earlier pictures are available for reference it is beneficial to let the encoder decide which one(s) to use for reference instead of relying on the decoder to make that selection.
[YK]: A few points to discuss. G.1: Is it backward compatible to RFC 4585 to use one RPSI message to convey more than one reference picture? Would it be better to define a new message for this purpose if it is deemed useful?
[JS] I don't see anything in RFC 4585 that would prohibit including more than one reference picture in the "Native RPSI bit string defined per codec" field.
G.2: The concept applies to any video codec that supports multiple reference pictures, is not specific to HEVC, thus would it be better if this is discussed as a generic mechanism, similarly as the SPLI message included in draft-ietf-payload-rtp-h265-01 but dropped later for the very reason?
[JS] This would be an option but I don't think it's needed since RPSI in RFC 4585 is already generic enough to govern this also for other codecs.
G.3: On the usefulness of this functionality seems to be unclear to me either. If the last correct picture before an error occurs is known, it is also known that the reference pictures of the last correct pictures are correct, thus the encoder can anyway choose to use one or more of these pictures, when available, for encoding the next picture. Thus, it might be sufficient just to slightly change the semantics of the RPLI message, to say that the encoder can choose to use one or more of the identified picture and its reference pictures for encoding the next picture.
[JS] The strength of RPSI in my opinion is that it can be used to recover from errors that is not a simple loss-of-a-NAL-unit-error (for this case generic NACK,  SLI and PLI are probably better suited). An RPSI message can be used to try to recover from any type of divergence between encoder and decoder without the need of a complete reset (i.e. through IDR picture). For such scenarios it might be very difficult to identify exactly which reference pictures are still available (and which of them are correct, as addressed by suggestion H. below) and it would be dangerous to assume that all reference pictures of one specific reference pictures will be available.

[YK]: Firstly, in my understanding, the decoder picture hash SEI message is very useful for debugging in development of encoder and decoder implementations, as used by the JCT-VC for the development of the HEVC reference software, and it is intended for use with that purpose. Using of it for error resilience in communications is an overkill to me. For example, you would need 48 bytes to signal the MD5 hash for each picture.
[JS] It would still be optional to include those 48 bytes of MD5 info, and if they can be used to detect errors that would otherwise not be detected, then 48 bytes per reference picture does not sound much compared to the typical bitrates for video streams.
Secondly, it is architecturally incorrect to couple this with the RPSI feedback message, as the picture identified is supposed to be correctly decoded by the decoder and the purpose of the message is to request the encoder to use that picture for encoding of the next picture,
[JS] You are right that the identified picture(s) is supposed to be correct. The proposed usage of MD5 provides a mechanism for verifying that a picture is actually correct.
thus why would you even indicate a picture that is incorrect using an RPSI feedback message?
[JS] See further motivation at the top of this email. That information can be used to identify incompatibility issues for immediate actions (change configuration) and long-term actions (identify non-conforming encoders/decoders). In scenarios where Full Intra reset is not feasible an encoder may use this information to perform gradual decoder refresh until the encoder and decoder have been verified to have identical reference pictures in their buffers.
Best Regards Jonatan
-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 30 april 2014 20:29
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear All,

Remaining open issues:

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, April 30, 2014 10:34 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org/>.

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Mon, 19 May 2014 16:48:34 +0000

Hi Jonatan, All,

On item C: In that example, you report three set of consecutive CTB losses, (2,2), (6, 2), and (10, 2). The semantics can only be interpreted this way, no?

On item D: It is up to the receiver to decide how much data is reported to be lost regardless of how much it actually received, but whatever it reports, it should report something that is accurate enough for the sender side to make meaningful reaction.

On item G: I'd like to see some opinions from others, particularly those who have been deeply involved in RFC 4585 to comment on the G.1 and G.2. On G.3, the wording "the encoder can choose to use one or more of the identified picture and its reference pictures for encoding the next picture" does not at all assume that all reference pictures of one specific reference pictures will be available, it just means that use whichever from that set when available.

On item H: Assume that the MD5 support as part of RPSI is specified, once a mismatch due to non-conformation encoder/decoder implementations is known, you still need people to get involved to debug and to see what is the reason of the mismatch, and you cannot assume the buggy implementations to be so powerful to super-intelligently figure out what is wrong with the implementation itself and then do a proper action to continue the session. Rather, in practice, I would think the signalling does not change anything to the session itself. For the codec implementation debugging purpose itself, encoders are always free to send the MD5 hash in the decoded picture hash SEI messages and decoders are free to parse and compare it, I don't see a need for yet another standardize effort for that.

BR, YK

From: Jonatan Samuelsson [mailto:[email-masked]]
Sent: Monday, May 19, 2014 6:29 AM
To: Yago Sanchez
Cc: [email-masked]; Wang, Ye-Kui
Subject: RE: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear Ye-Kui, Yago, all,

Best Regards Jonatan

From: Yago Sanchez [mailto:[email-masked]]
Sent: den 17 maj 2014 12:47
To: Jonatan Samuelsson
Cc: [email-masked]<mailto:[email-masked]>; Wang, Ye-Kui
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Hi Jonatan,

Best regards,
Yago

Hi Jonatan,

BR, YK

In 8.2:

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 30 april 2014 20:29
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear All,

Remaining open issues:

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, April 30, 2014 10:34 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Abstract:

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Wed, 21 May 2014 01:38:25 +0000

Dear All,

The following group of persons had a conference call today (May 20, 2014, 11:00am-12:30pm PDT), with the goal to resolve the remaining open issues for this payload format, such that it can be finalized as soon as possible due to the urgent need of the HEVC payload format for the deployment of HEVC:

-          Magnus Westerlund

-          Stephan Wenger

-          Jonatan Samuelsson

-          Yago Sanchez

-          Ye-Kui Wang

The discussed list of open issues are:

1)      Ticket#12 (rtp-h265): The max-fps parameter

2)      On source-specific sprop parameter sets

3)      On support of a future profile defined as the intersection of more than one profile

4)      On recv-sub-layer-id and using of sprop-vps for session negotiation

5)      On the SLI feedback message: in CTB raster scan order or in tile scan order, for both FIRST and NUMBER

6)      On the RPSI feedback message

a.       Signalling of more than one reference picture to be used for encoding the next picture

b.      Support of integrity check

What we agreed among the participants on the above issues are as follows:

1)      Ticket#12 (rtp-h265): The max-fps parameter

With the clarifications in the latest email from Geir Sandbakken, we agreed that it would be OK with the use of the MUST language as Geir suggested.

2)      On source-specific sprop parameter sets

We agreed to add a clarification sentence like: When conveyed in the "a=fmtp" line of SDP for a particular payload type, the parameters "sprop-vps", "sprop-sps", and "sprop-pps" MUST be applied to each SSRC with the payload type.

3)      On support of a future profile defined as the intersection of more than one profile

We agreed to have the support, but clearly document the intention of such support, to avoid possible confusions, such as requiring encoders to support all possible intersections as discussed between Jonatan and myself. Also try to clarify the implications of when the profile related parameters are used with different direction attributes. Magnus kindly agreed to draft some text.

4)      On recv-sub-layer-id and using of sprop-vps for session negotiation

We agreed to allow the use of sprop-vps for session negotiation, but require profile, tier, level info to be present in the SDP answer. Compared to completely removing this mechanism, this approach supports selection of one of the multiple operation points, particular when more than one of the operation points comply with the some profile, tier, and level.

5)      On the SLI feedback message: in CTB raster scan order or in tile scan order, for both FIRST and NUMBER

We agreed to keep the current method (i.e. in CTB raster scan order), but clarify a bit to make it clearer, e.g. including mentioning in some cases with tiles multiple messages might be needed for reporting the loss of one slice.

6)      On the RPSI feedback message

a.       Signalling of more than one reference picture to be used for encoding the next picture

b.      Support of integrity check

We agreed to investigate both a and b above in a separate document in a generic way in the context of avtext. Jonatan to initiate the document. For this payload format document, we agreed to add the parameter "include-dph", as Jonatan suggested to the mailing list. For convenience, the semantics of this parameter is provided is as follows:
include-dph:
This parameter is used to indicate capability and preference to utilize or include Decoded Picture Hash (DPH) SEI messages (See Section D.3.19 of [HEVC]) in the encoded video sequence. The value is a comma separated list of hash types that is supported or requested to be used, each hash type provided as an unsigned integer value (0-255), with the hash types listed from most preferred to the least preferred. Example: "include-dph=0,2", which indicates the capability for MD5 (most preferred) and Checksum (less preferred). If the parameter is not included or the value contains no hash types, then no capability to utilize DPH SEI messages is assumed. Note that DPH SEI messages MAY still be included in the encoded video sequence even when there is no declaration of capability to use them.

Comments and suggestions are welcome! Unless otherwise concluded, we would implement text changes for the above issues to the document. Thanks!

BR, YK (representing the above list of participants to the call)

From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: Monday, May 19, 2014 9:49 AM
To: Jonatan Samuelsson; Yago Sanchez
Cc: [email-masked]
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Hi Jonatan, All,

BR, YK

Dear Ye-Kui, Yago, all,

Best Regards Jonatan

Hi Jonatan,

Best regards,
Yago

Hi Jonatan,

BR, YK

In 8.2:

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 30 april 2014 20:29
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Dear All,

Remaining open issues:

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, April 30, 2014 10:34 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [payload] I-D Action: draft-ietf-payload-rtp-h265-03.txt

Abstract: