Subject: [payload] Adding signaling of parallel decoding capabilities to the HEVC/H.265 payload specification

From: Rickard Sjöberg <[email-masked]>
Date: Mon, 22 Jul 2013 16:02:26 +0000

Dear all,

We lack a feature in the H.265/HEVC payload specification which we would like to address. We see that the number of cores used in devices that decodes video is increasing. New high-end computers, tablets and phones typically have four cores or more. This trend towards more cores in devices seems to continue, albeit slow.

With this in mind, it would be good to address multi-core architectures in the session negotiation of HEVC, especially for Offer/Answer, as a part of the RTP payload specification. Let me give you an example: Assume that a multi-core device is capable of decoding 720p30 sequentially. This is all well, but assume further that since the device has multiple cores it is actually capable of decoding 1080p30 given that the bitstream is made parallel by using e.g. slices, tiles, or wavefronts. It would then be beneficial if the decoder could express that an encoder could either send a parallel stream up to 1080p30 or a sequential stream up to 720p30. As the draft stands now, the decoder would have to declare 720p30 as the maximum since there are no guarantees that the stream is parallel.

This is not yet addressed in the payload specification draft. The current draft includes the spatial-segmentation-idc parameter for HEVC that conveys the parallel property of a stream. But in the draft it is only used to describe the stream and not for specifying receiver capabilities. What we propose is to use spatial-segmentation-idc as base also for expressing receiver capabilities.

The basics of our suggestion is that each decoder can optionally declare higher decoding capabilities given that the spatial-segmention-idc value of the received stream have a given value (or higher). The decoding capabilities are expressed as tier and level for the profile defined by the payload type and any extended encoding capabilities expressed using appropriate parameters, such as the max-ls and max-lps parameters. Multiple such declarations can be given to express different capabilities at different levels of spatial-segmentation-idc and to sufficiently express the capability space for each spatial-segmentation-idc value.

The set of decoding capabilities expressed, including the basic level (spatial-segmentation-idc=0) expressed using the existing level + extension parameters, are then provided to the peer which can then choose how to encode the video stream to meet the decoder capabilities. As this parallelism decoding capability information is receiver related and declarative, the information actually fits SDP Offer/Answer well.

Given the 1080p30/720p30 example above, the decoder could use any of these examples to express its parallel capabilities:
// Level 3.1, but if spatial-segmentation-idc >= 16 then the receiver is also capable of Level 4
a=fmtp:98 profile=High;Level=3.1;dec_parallel_cap={16;Level=4}
// Level 3.1, but if spatial-segmentation-idc >= 16 then the receiver is also capable of Level 3.1 with lps equal to 1920x1080 and max-ls equal to 1920x1080x30:
a=fmtp:98 profile=High;Level=3.1;dec_parallel_cap={16;max-ls=62208000;max-lps=2073600}

A more complicated example could be:

a=fmtp:98 profile=High;level-id=2.1;max-recv-level-id=3.1;dec-parallel-cap={8;max-ls=46080000,16;max-ls=62208000;max-lps=2073600,16;level-id=4}

This example describes the configuration of payload type 98 where the agent declares extended decoding capabilities given some stream requirement related to parallel decoding. Three different points beyond the configuration for sequential decoding are given:

1)      No requirement on min_spatial_segmentation_idc:
Using Level 3.1 which supports 720p30 (max-recv-level-id overrides the symmetric level-id).

2)      min_spatial_segmentation_idc >=8
Level 3.1 extended with max-ls =46080000 is supported (enables 720p50).

3)      min_spatial_segmentation_idc >= 16
Two different capability points are provided:

a.       Level=3.1 with max-ls=62208000 and max-lps=2073600 (enables 1080p30)

b.      Level 4.0.

The attachment provides draft text changes to the payload specification. We believe it is important to address multicore devices and that the text changes provides what is necessary in a compact and straightforward way.

Best regards
Rickard Sjoberg, Bo Burman, and Magnus Westerlund