专利摘要:
ENCODING CONCEPT THAT ALLOWS PARALLEL PROCESSING, TRANSPORT DEMULTIPLEXER AND VIDEO BITES FLOW A raw byte sequence payload that describes an image in pieces, subflows or WPP boards and encoded using binary arithmetic coding context adaptive, it is subdivided or cut into tranches with the continuation of the probability adaptation of context adaptive binary arithmetic encoding beyond the limits of the tranche. Through this measure, the limits of the tranche additionally inserted into pieces, subflows or WPP plates do not reduce the efficiency of the entropic coding of these elements. On the other hand. however, the tranches are smaller than the original pieces, subflows or WPP plates and can be transmitted earlier, that is, with less delay than the original uncut entities, that is, the pieces, subflows or plates WPP. According to another aspect, which can be combined with the first aspect, the subflow marker NAL units are used within the sequence of NAL units of a video byte stream to allow a transport demultiplexer to assign chunk data within the NAL units to the corresponding ones (...).
公开号:BR112014017915B1
申请号:R112014017915-8
申请日:2013-01-21
公开日:2021-03-16
发明作者:Thomas Schierl;Valeri GEORGE;Karsten GRÜNEBERG;Heiner Kirchhoffer;Anastacia Henkel;Detlev Marpe
申请人:Ge Video Compression, Llc;
IPC主号:
专利说明:

[001] The present invention relates to coding concepts that allow parallel processing, as in HEVC wrapping, a transport demultiplexer and a stream of video bytes.
[002] Parallel encoder and decoder is very important due to the processing requirements increased by the HEVC standard, as well as the expected increase in video resolution. Multi-core architectures are becoming available on a wide variety of modern electronic devices. Consequently, efficient methods are needed to allow the use of multi-core architectures.
[003] The encoding or decoding of LCUs occurs in a scan analysis, whereby the CABAC probabilities are adapted to the specificities of each image. There are spatial dependencies between adjacent LCUs. Each LCU depends on its left neighbor, above, above left and above right, due to different components, for example, motion vector, prediction, intra-prediction and others. In order to allow parallelization in decoding, these dependencies must normally be interrupted or are interrupted in current technology applications.
[004] Some parallelization concepts have been proposed, namely wavefront processing, which use entropic pieces [3], parallel wavefront processing (WPP) operations, using subflows [2] [4], [11] or tiles [5]. The latter does not necessarily have to be combined with front-end processing to allow parallelization in decoding or encoding. From this point of view, tiles are identical to WPP subflows. Our initial motivator for continuing to study the concept of entropic bits is to implement techniques that reduce the loss of coding efficiency, thus reducing the burden on the flow of bytes for parallelization approaches in coding and decoding.
[005] For a better understanding, particularly the use of LCUs, it is necessary to look first at the structure of H.264 / AVC [1].
[006] A video sequence encoded in H.264 / AVC consists of series of access units that are collected in the flow of the NAL unit and use only one set of sequence parameters. Each video sequence can be decoded independently. An encoded sequence consists of a sequence of encoded images. An encoded image can be an entire image or a single field. Each image is divided into fixed size macro-blocks (in HEVC [5]: LCUs). Various macro-blocks or LCUs can be merged into one piece. An image is, therefore, a collection of one or more pieces. The purpose of this data separation is to allow independent decoding of samples in the image area, which is represented by the piece, without using data from other pieces.
[007] A technique that is often called “entropic pieces” [3] is a fractionation of the traditional piece into additional sub-pieces. Specifically, it means forming pieces of entropic encoded data from a single piece. The arrangement of entropic pieces in a piece can have different varieties. The simplest is to use each row of LCUs / macroblocks in an image as an entropic piece. Alternatively, the separate columns or regions can be used as entropic pieces, which can be interrupted and articulated with each other, e.g. ex. piece 1 in Fig. 1.
[008] An obvious objective of the entropic chunk concept is to allow the use of parallel CPU / GPU and multicore architectures, to improve the decoding process time, that is, to speed up the process. The current chunk can be divided into partitions that can be analyzed and reconstructed without reference to other chunk data. Although some advantages can be achieved with the entropic chunk approach, some penalties do emerge.
[009] The concept of entropic pieces has been extended to subflux wavefront processing (WPP), as proposed in [2], [10], [11] and partially integrated in [5]. Here the subflow repeating scheme is defined. Which have an entropy state initialization improved by line compared to entropic pieces.
[010] The tile concept allows the separation of information from the image to be encoded, while each tile has its own scan analysis order. A tile is defined by a common structure, which is repeated in the image. A tile can also have a certain column width and line height in terms of LCUs or CUs.
[011] Tiles can be coded independently and can also be coded so that they do not need to be joined with other tiles, so that these decoder frames can process tiles from an Access Unit completely, or at least for some steps of coding operation, independently, that is, entropic coding and transformation coding.
[012] Therefore, a tile allows you to run tile encoders as well as decoders, totally or partially independently, in a parallel way upwards, in the latter case, p. ex. filtering stage of the HEVC codec.
[013] To make full use of the parallelization techniques in the capture, encoding, transmission, decoding and presentation stream of a video communication system or similar systems, the transport and access of data between the communication participants is an important step and time consuming for the entire end-to-end delay injection. This is especially a problem if you use parallelization techniques, such as tiles, subflows or entropic pieces.
[014] WPP subflow data approaches imply that the encoded data of the partitions, if processed, have no data locality, that is, a single thread decoding the Access Unit, potentially has to jump large parts of memory to access data from the next WPP subflow line. A multithreaded decoding system has to wait to transmit certain data, that is, WPP subflows, to work in a completely parallelized way, exploiting wavefront processing.
[015] In the video stream, allowing higher resolutions (Full-HD, QUAD-HD etc.) leads to larger amounts of data that have to be transmitted. For time-sensitive scenarios, so-called low use-case delays, such as video conferences (<145 ms) or game applications, (<40 ms) very low end-to-end delays are required.
[016] Therefore, transmission time becomes a critical factor. Consider the ADSL charging connection for a video conferencing application. Here, the so-called random access flow points, usually these refer to images I, will be the candidates to cause a strangulation during transmission.
[017] HEVC allows the so-called Wave Front processing, as well as the processing of tiles on the encoder side and on the decoder side. This is allowed by the use of entropic pieces, WPP subflows or even a combination of these. Parallel processing is also allowed by the parallel encoding and decoding of tiles.
[018] In the case of “non-parallelization objective”, the data of an entire piece would be delivered at once, thus being the last CU of the pieces accessible by the decoder if it has been transmitted. This is not a problem, if there is a single chained decoder.
[019] In the case of multiple chaining, if multiple CPUs or cores can be used, the decoding process would, however, like to start as soon as the encoded data reaches the Wavefront decoder or tile decoder threads.
[020] Therefore, it would be favorable to have concepts at hand that allow to reduce the coding delay in parallel processing environments with less serious reductions in coding efficiency.
[021] Correspondingly, the present invention aims to provide a coding concept, a concept of transport demultiplexing and a stream of video bytes, which allow for much more efficiency, a low delay encoding in parallel processing environments.
[022] This object is achieved by the question of the attached independent claims.
[023] According to a first aspect of this application, a raw byte sequence payload that describes an image in pieces, WPP subflows or tiles and encoded using context adaptive binary arithmetic encoding is subdivided or cut into plots with the continuation of the probability adaptation of context adaptive binary arithmetic encoding beyond the boundaries of the plot. Through this measure, the limits of the parcel additionally introduced into pieces, WPP subflows or tiles do not reduce the efficiency of entropic coding of these elements. On the other hand, however, the portions are smaller than the original pieces, WPP subflows or tiles and can respectively be transmitted earlier, that is, with less delay than the original uncut entities, that is, the WPP pieces, subflows or tiles.
[024] According to another aspect, which can be combined with the first aspect, the subflux marker NAL units are used within the sequence of NAL units of a video byte stream to allow a transport demultiplexer to allocate data from pieces within the NAL units to the corresponding subflows or tiles, in order to be able, in parallel, to serve as a multiple chaining decoder with the corresponding subflows or tiles.
[025] Advantageous implementations are the subject of dependent claims. In addition, the privileged versions of the present invention are explained in more detail below with respect to the figures:
[026] Fig. 1 shows a schematic showing the possible compounds of entropic pieces;
[027] Fig. 2 shows a schematic showing three tiles spread over three pieces;
[028] Fig. 3shows the schematic illustrating an interim example of trances of a cyclic interim scheme of plots of variable length in four;
[029] Fig. 4 shows a schematic illustrating an encoding, segmentation, interleaving and decoding of entropic piece data;
[030] Fig. 5 shows a schematic illustrating an interim example of trances in the cyclic interim scheme of trances of variable length in four, always using marker codes and the disclosure of the current piece data over multiple NAL units. Bookmark codes are used, even if the partition is not present. This can be further improved by using a plot identifier, after the marker, indicating the plot number. This dispenses with the need to always send a bookmark, as required for cyclic mode;
[031] Fig. 6 shows a table of pseudo-codes, illustrating unit syntax;
[032] Fig. 7 shows a table of pseudo-codes, illustrating a sequence of parameter set syntax;
[033] Fig. 8 shows a table of pseudo-codes, illustrating a RBSP syntax of the Low Delay Pieces layer;
[034] Fig. 9 shows a table of pseudo-codes, illustrating a chunk header syntax;
[035] Fig. 10 shows a table of pseudo-codes, illustrating a Subflow marker syntax;
[036] Fig. 11 shows a schematic illustrating an example for a single encapsulation of entropic chunk data. (AF is the MPEG-2 TS Adaptation Field);
[037] Fig. 12 shows a schematic illustrating an example for a Single ES encapsulation of entropic chunk data;
[038] Fig. 13 shows a schematic illustrating another example for a Multi-ES Packaged encapsulation of entropic chunk data;
[039] Fig. 14 shows a block diagram schematic, illustrating a single ES Transport demultiplexer; and
[040] Fig. 15 shows a block diagram schematic, illustrating a Transport demultiplexer for multi-ES;
[041] Fig. 16 shows a schematic block diagram, which illustrates an encoder;
[042] Fig. 17 shows a block diagram schematic, illustrating a decoder;
[043] Fig. 18 shows a flow chart of steps performed by the decoder; and
[044] Fig. 19 shows a schematic illustrating an example for multi-ES, using RTP.
[045] To reduce the time that a parallel decoder chain can start and end its data from an image, the versions below use a segmentation of the data, structured for parallelization, such as data from one or more tiles or data from an or more WPP subflows in small plots through a low delay interim approach.
[046] Therefore, the encoder can deliver data, corresponding to a particular set of LCUs or at least an aligned portion of bytes of a subflow or tile or parts thereof in the form of a portion to the decoder through the transmission path of the encoder to the decoder.
[047] Since the plots are smaller than the entire WPP or tile subflow and / or can be adapted to the current maximum transfer unit (MTU) of the transmission path, so that the plots of multiple WPP subflows or tiles can be arranged in a transfer unit between the encoder and decoder, before the completion of the complete access unit, decoding on the decoding side can be initiated significantly earlier than using a sequential transmission of the WPP subflows or complete tiles of a Access Unit.
[048] This obviously results in a faster transmission of the parcels and an earlier start of a parallel decoding process in the decoder. The approach can also be applied beyond image limits, in case the next image piece (s) or entropic piece (s) can already be decoded, p. eg, in a wavefront mode, based on the knowledge that the information required to decode an entropic piece of an image that follows due to availability and inter-image references. These data already decodable from an image that succeeds in the decoding order can be derived from the maximum length of the allowed / signaled motion vector or from the additional information in the flow that indicates the dependencies of data pieces for the image / images precedents, or a fixed referencing scheme, indicating the position used signaled in a fixed sequence position, as a set of parameters.
[049] An image can be encoded with an entropic piece per line (s) (LCU) from the larger coding unit or using WPP subflows or even a combination like a WPP subflow per line, which can still be included in a separate Entropic Piece . This type of data structure is necessary to make use of the Wavefront processing technique on the decoder side. Or tiles can be used to allow parallel processing.
[050] During the encoding process, the bit stream of each piece, which contains the data from WPP streams or tiles, can be divided into portions of variable size to correspond to the maximum size of the transfer unit, between encoder and decoder. Then, the resulting parcels are interleaved and can be passed on to the transmission and placed in MTU size packages.
[051] To allow processing on the decoder side, before or after each plot, a marker code can be inserted. A marker code suitable for HEVC can be “0x00 00 02”, which would even pass the start code emulation prevention. Upon receipt of a package that includes multiple parcels, the receiver or decoder can analyze the stream of bits currently contained during the start code emulation prevention process so as not to require an additional analysis step. There may, for example, be two ways of identifying the parcel. There can always be a cyclical arrangement of the parcels, starting with the parcel with parcel_id (parcel identifier) equal to 1 for the parcel with parcel_id equal to n. This can save the signaling data for the second general method. An alternative method can be a specific header after the marker, indicating the parcel_id, p. ex. as a bit value of 8.
[052] Deinterleaving of interleaved parcel data can be applied based on knowledge of the number of parcels per package, which can be an NAL unit package. Therefore, there may be additionally a mapping of WPP subflows or tiles to plots. This mapping can be implicitly derived from the number of tiles / number of WPP subflows, or it can be signaled directly at the PLC. The mapping is important for the deinterleaving process, so that the data of certain WPP subflows or tiles can be identified and served in front of the wave or chaining of a parallel decoder in charge of decoding the WPP or tile subflow in question.
[053] To inform the decoder about the use of the interleaving scheme for the low-delay encapsulation, there may be a low-delay_band in the NAL unit header.
[054] Another way can be interleaving and deinterleaving over the transport layer, that is, outside the decoding process, perhaps in RTP [8] [9] [0154] or MPEG-2 Transport Flow layer [7].
[055] Therefore, a header can be placed in front of the package, indicating the presence of a parcel through a flag including size information in bytes per parcel present. Since the transport layer is decoupled from the decoding process, there may be no need to integrate a marker code, as the additional information from the transport layer has to be removed in any case before passing that data on to the decoder. The transport layer then also records data for the delivery of byte streams to the decoder.
[056] A variable length header can be used in an extra multiplexing layer. This multiplexing layer can also be part of a codec and can be introduced before the current access to Raw Byte Sequence Data (RBSP) in the decoder. A header diagram can be found in Fig. 3. But there can also be a header directly in front of each plot, indicating the length, as well as, of its indicator. Where there is still a need to map the indicator to byte flow structures as has been stated before.
[057] The size of the plot can also be constant, p. ex. x bytes per parcel. This results in a simple multiplexing scheme, as shown in Fig. 4.
[058] The constant size of the segments can cause a problem at the end of the byte flow due to their variable size.
[059] Two general solutions are possible. The first is a generation of cyclic x-byte segments (usually the bitstream representation of the chunk is aligned with the byte) and the control of byte consumption by each decoder engine, that is, the decoder discovers the complementation of an entropic piece or including a marker code.
[060] The second method is the signaling of parcel lengths, if the parcels are of variable size in a header, as shown in the figure.
[061] The segment size and the interleaving mode can be signaled in a SEI message or in the PLC.
[062] The transmission scheme is shown in Figure 4.
[063] Another interesting method is the use of completion codes or marker codes at the end of the set of parcels in the package, such as NAL or piece parcel. In this case, segments of variable length are possible, requiring a complete analysis of the flow of bytes. To limit memory access here, this additional analysis process for multiplexing can be combined with the boot code emulation prevention analysis, required as a first step before accessing the RBSP data contained in an NAL unit. This type of marker scheme is shown in Figure 5.
[064] The idea here is to split an upper level structure, such as a current piece, entropic piece or similar, in its contained lower level data structure, such as WPP subflows or tiles, while merging the data into plots. These plots, each belonging to a low-level structure, p. ex. a WPP subflow or a specific tile, are interleaved in a low delay package, which can be a specific NAL unit, an NAL unit with additional signaling via a low delay interleaving flag or even a low piece or header weight indicating the low delay interleaving approach using a flag or type of chunk, as shown for “NAL unit # 1” in the figure, thus the decoder informed to apply a recording function to a chaining decoder “ single ”, which is using sequential processing of the parcels in the original / deinterleaved order in the decoder. To split data from a current chunk as interleaved parcels over multiple packets to gain the low delay characteristic, a transport layer can fragment the NAL unit that contains the low delay interleaved data for maximum MTU size network packets. The fragmentation of the current piece data into multiple NAL units can also be directly applied by the coding layer, thus there is a need to signal this type of NAL unit that contains the continuation of a piece, as shown in Fig. 5 for “NAL unit #two". To detect the completion of data merged in multiple packets, such as NAL units. There may be a need for a specific completion code, as it is also shown for “NAL unit # 2” in the figure or a flag that indicates the completion on the NAL piece or header.
[065] In the case of losing NAL packets, there is also a need to detect losses. This can be applied by additional information in the header, p. ex. the header of a low-weight chunk, such as the first MBs of the contained parcels or just a specific parcel # 1. Having information like the deviations for the WPP subflows or the current plot size, someone can also use these size values (deviation values for a specific WPP subflow or tile) to do a sanity check after receiving the NAL unit with the completion code and the preceding NAL units.
[066] That is, as described, the parcels can be packaged in packages 300 so that each package 300 comprises a T # portion of each WPP subflow or image tile, or a subset of the WPP subflows or image tiles (because , for example, a certain sub-flow WPP or tile has already been completely transported through the previous packages), arranged in an order # defined among the sub-flows WPP or tiles, each package comprising a header 302 with the disclosure of the positions and / or lengths of the T # plots packaged in the respective packet 300, or markers 304 separating the T # plots from each other within the respective packet 300, in which the decoder can be configured to, when receiving the raw byte sequence payload, use the information understood by headers 302 or markers 304, in order to access the parcels within the packages. The 300a packets comprising the first - according to the defined order among the WPP subflows or tiles - portions of the WPP subflows or image tiles, may comprise a 306 indicator of low delay characteristics, and the 300b packets comprising the second or subsequent - according to the order defined among the WPP subflows or tiles - T # portions of the WPP subflows or tiles of the image, may comprise a continuation indicator 308. The packages 300 can be NAL units or pieces
[067] Next, an example is provided to signal the syntax and semantics for the interleaving of low delay in plots.
[068] However, the division of plot data, such as data from a WPP or tile subflow, can also be applied at the chunk level or below, as stated above.
[069] Now, an approach is presented, which can be combined with analysis to prevent boot code emulation to reduce additional processing steps. Therefore, an interleaving at the RBSP level of the HEVC codec is applied.
[070] A plot can be seen as dividing RBSP data into sections to be merged into the payload section of the NAL unit for accessing low delay data. The completion of a parcel can be indicated by the code 0x000002 and can be followed by a parcel identifier 8bit parcel_id. The parcels can be interleaved cyclically, so that the final code of the parcel is not followed by the parcel_id, which is implicitly derived. The Rbsp data in a single plot corresponds to tile data, subflow data, chunk data or entropic chunk data.
[071] In the syntax of the NAL unit, two modes for low delay interleaving can be allowed as indicated by the “low delay encapsulation_band”, that is, the cyclical arrangement of the parcels, as well as an indication of the parcel through an additional identifier “Parcel_id” that follows the marker encoder through a flag, such as the “cyclic_band of low delay” in the header of the NAL unit. These two flags can also be present in the Sequence Parameter Set or even in the APS. For the cyclic plot provisions, there may still be a need to know the number of plots during the analysis, as provided in the CLP as “núm_baixo_atraso_parcelas”.
[072] In the NAL unit, the interleaved “LD_rbsp_byte” s are read by the analyzer, a recorder for the current sequential RBSP order in the last para-cycle in the NAL syntax: para (i = 0, i ++, i <num_under_parallel_parts) {para ( j = 0, j ++, j <NumBytesEmRBSP [i]) {rbsp_byte [NumBytesEmRBSP ++] = LD_rbsp_byte [j] [i]}
[073] There may also be an explicit signaling in the CLP or APS for a fixed size of cyclically arranged plots as indicated in the “Baixo_atraso_parcela_comprimento_menos1”. The latter was not used in the example of the NAL unit syntax, but it is straightforward if it has a packaging as shown in Fig. 4 in at least In the syntax of the NAL unit in Fig. 6, a packaging as shown in Fig. 5 and discussed above it was the basics.
[074] To enable this feature of plot interleaving over multiple packages, such as pieces and / or NAL units, there may be a requirement for a global temporary memory, such as the LD_rbsp_byte network for the parcels, in order to have repeated access to the data RBSP of NAL units already received. To allow error resilience, after receiving a completion code, or if the sum of the number of bytes received for a parcel is equal to the parcel size, which can be derived from the deviation values as provided for the parcel data contained , P. ex. of the data related to the respective WPP or tile sub-flow, of which the parcel in question is part.
[075] An important condition for WPP subflows arranged in interleaved low-delay plots is that a n + 1 plot can only access data from plot n, which has already been provided in plot n and is already saved or available in the decoder.
[076] The RBSP syntax of the Low Delay Piece layer for reordering / deinterleaving at the piece level can be designed as follows. In particular, the syntax must, in this case, have almost the same behavior as the layer of the NAL unit, but the reordering has to be defined at the chunk level. Fig. 8 shows an RBSP syntax for the Low Delay Pieces layer.
[077] In the case of using the header of the piece to package the interleaved parcels, there may be a requirement to indicate, at the codec level, if receiving a new piece, the non-restoration of the CABAC state, since the entropic encoding of parcels of, p. ex. a WPP subflow, should not be interrupted. Do not replace the CABAC in a piece is indicated as "no_cabac_reset_bandeira" in the piece's header. The piece header shown is suitable for low-delay pieces, so the entropy_pedaço characteristics must be presented. A corresponding piece header syntax is shown in Fig. 9.
[078] The transport layer allows the optimization of the organization of data forwarded to the decoder unit (s) based on the fact that a number of subflows / tiles / plots (in the transport layer, we assume an abstract entity which can be represented by a subflow, a tile, part of a subflow or tile, or a part of the byte stream that has an identical function, that is, it allows parallel decoding or the gradual renewal of the decoder) can be processed in the layer coding independently of each other. One possibility is to start sending parcels in parallel to several decoding units with minimal delay. The flow of bytes consists of a sequence of NAL units which are the smallest items that can be individually handled in the transport layer. Consequently, the following methods of handling in the transport layer are based on subflows / tiles / plots, which are contained in a separate piece or in units of NAL entropic pieces.
[079] The transport layer should also optimize decoder performance and error resilience based on the fact that the encoding layer uses a gradual decoder renewal. One option is to dispense with irrelevant parts of the byte stream if there are earlier parts of the byte stream that were not received correctly, p. ex. due to transmission errors, or that were not even received, p. ex. due to a switch between transport channels.
[080] To allow this exploration / optimization, different information is signaled in the transport layer.
[081] The general lateral information is signaled, using descriptors:
[082] - The number of subflows / tiles, where "1" means that there is only one flow / tile that contains the entire video image;
[083] - Information common to all subflows / tiles, p. ex. if all subflows / tiles are the same size or if the buffer requirements are the same;
[084] - Individual information about each subflow / tile, p. ex. if the subflows / tiles have a different size or if your temporary memory requirements differ;
[085] - Number of steps for the gradual renewal of the decoder, where "1" means that the gradual renewal of the decoder is not used;
[086] - A flag indicating whether these subflows / tiles allow for low delay parallel processing;
[087] If the number of subflows / tiles> 1, the elements of the syntax are inserted in the flow before each data block that contains a certain subflow / tile. These syntax elements follow the syntax of the NAL unit, but use a single type of NAL unit that is not used by the coding layer (eg nal_unit_type = 0x19 or nal_unit_type = 0x1F), which is now called sub-flow markers .
[088] These syntax elements are used as markers and carry information about the data block that follows at least one data field that identifies the subflow / tile.
[089] If the number of steps of gradual renewal of the decoder> 1, these syntax elements also carry a flag indicating whether the subflow / tile is intra-coded (allows the gradual renewal of the decoder).
[090] A corresponding syntax is shown in Fig. 10. The following restrictions can be applied: prohibited_zero_bit must be equal to 0; nal_ref_bandeira must be equal to 0; nal_unit_type must be equal to 0x19; subfluxo_ID: counter value starting with 0 for the first piece that belongs to an image, incremented by each additional piece or entropic piece that belongs to the same image; é_intra: if ‘1’, the following NAL unit contains an intra-encoded piece or an intra-encoded entropic piece.
[091] A method for encapsulating the video stream in a transport multiplexing is shown in Fig. 11, where each piece or entropic piece is separately transported in an entire number of transport stream packages. If the size of the payload does not exactly match the bytes available in the TS packets of fixed size, the last TS packet contains an adaptation field.
[092] Note that a similar behavior of the Elemental Flow of MPEG-2 Transport Streams can also be provided by an RTP session or an RTP flow of the Real-Time Transport Protocol, as illustrated in Fig. 19. In RTP [ 0149], an RTP Flow (identified by the type of medium and type of payload as indicated in SDP [0153]) may be contained in its own RTP session, in which an RTP Session is identified by the network (IP) address, the port (UDP), as well as the originator identifier (SSRC). A media session as indicated in the SDP can contain multiple RTP sessions, each having a different media type. But it is also possible to carry the same medium stream (eg video) in different RTP streams, where the RTP streams can be contained in the same RTP session (analogous to 1. below) or can be contained in your own RTP sessions ( analogous to 2. below). Fig. 19 illustrates case 2.
[093] The RTP payload formats [0150] [0154] have a decoding order number (DON), which allows you to retrieve the decoding order of the NAL units on the receiver if they are intentionally transmitted outside the decoding order to effects of error resilience, as described in [0150] [0154]. Additional MKR markers are therefore not required. In the case of transport parcels of the WPP subflows or tiles in the order in which they are available from the encoding processes, the DON can also be used to retrieve the parcel order before supplying them to the single decoder.
[094] But in this case, an additional delay in the decoder would be introduced due to the separate deinterleaving process before the decoding process. The system described here can provide the encoded parcels directly for the decoding processes of the different WPP subflows or tiles, while the data arrives at the receiver. The parcel identification associated with a WPP or tile subflow can be derived by the piece address in the piece segment header of the piece segment and the order of transmission of the packages as indicated by the RTP sequence number in the RTP header.
[095] In this scenario, DON is used for backward compatibility, that is, for decoders that do not provide the improved ability to decode portions of WPP subflows or tiles sent out of the decoding order when they arrive. The sending of data from parcels outside the decoding order is only applied in relation to the level of WPP subflows or tiles, that is, in the transmitted data, the portions of a single WPP or tile subflow are transmitted in decoding order, the data being of the different WPP subflows or tiles are interspersed.
[096] There are two possible options: 1. All entropic bits and pieces are contained in the same elementary stream, that is, the same PID is assigned to all TS packets in that video stream; in the text that follows this method is called single ES encapsulation; 2. The different PIDs are assigned to pieces and entropic pieces of the same video byte stream; in the text that follows this method is called multi-ES encapsulation.
[097] Fig. 11 is valid for both options if the first option is considered a special case of the more general structure through the definition of the same PID for all ES.
[098] A more efficient way to encapsulate in a single ES is shown in Fig. 12. Here, a maximum of one image adaptation field is required.
[099] A more efficient way of encapsulating multiple ES is shown in Fig. 13. Here, adaptation fields are avoided; instead, another piece, p. ex. the tile placed in the next image, starts immediately in the same transport flow package.
[0100] In Fig. 14 a possible structure of the transport demultiplexer is presented for the encapsulation with a single elementary flow (ES) directed to a multi-chain decoder. The entropic piece in the figure can contain data from a specific WPP subflow or tile.
[0101] The Temporary Transport Memory (TB) collects data that belongs to a transport package and forwards it to the Multiplexing Temporary Memory (MB). At the output of the MB, the headings of the NAL unit are evaluated and the subflow markers are dispensed with, while the data carried in the subflow marker is saved. The data for each piece or entropic piece are stored in a separate Piece Temporary Memory (SB), from where they are pulled by a multi-thread decoder, as soon as a decoder chain is available.
[0102] In Fig. 15 a possible structure of the transport demultiplexer is presented for the encapsulation with multiple elementary streams directed to a multithreaded decoder.
[0103] The concepts outlined above are described again below, but in other words. The description below can therefore be combined individually with additional details from the description above.
[0104] Fig. 16 shows a general structure of an encoder according to a model of the present application. The encoder 10 could be implemented to be able to operate in a multi-threaded mode, that is, only single-threaded. That is, encoder 10 should, for example, be implemented using multiple CPU cores. In other words, encoder 10 could support parallel processing, but it does not have to. The coding concept of the present application allows parallel processing encoders to effectively apply parallel processing without, however, compromising the effectiveness of compression. Regarding the parallel processing capacity, similar statements are valid for the decoder, which is described later in relation to Fig. 17.
[0105] Encoder 10 is a video encoder, but in general encoder 10 can also be an image encoder. An image 12 of a video 14 is presented as input encoder 10 on an input 16.
[0106] Encoder 10 is a hybrid encoder, that is, image 12 is predicted in predictor 18, and the prediction residual 20 as obtained by a residual determinator 22, such as a subtractor, is subjected to a transformation , such as a spectral decomposition such as a DCT, and a quantization in a transformation / quantization module 24. A quantized residual 26 thus obtained is subjected to entropic encoding in an entropic encoder 28, namely the context adaptive binary arithmetic coding. The reconstructable version of the residual as available for the decoder, that is, the decanted and retransformed residual signal 30, is retrieved by a retransformation and requantization module 31, and combined with the prediction signal 32 of the predictor 18 through the combiner 33, resulting thus in a reconstruction 34 of the image 12. However, the encoder 10 operates on a block basis. Accordingly, the reconstructed signal 34 suffers from discontinuities at the limits of the block and correspondingly, a filter 36 can be applied to the reconstructed signal 34 to monetize a reference image 38, on the basis of which the predictor 18 subsequently predicts encoded images. As can be seen in dashed lines in Fig. 16, predictor 18 can, however, also exploit the reconstructed signal 34 directly without filter 36 or an intermediate version. In the case of image coding, filter 36 can be omitted.
[0107] Predictor 18 can choose between different prediction modes to predict certain blocks in the image 12. There may be a time prediction mode, according to which a block is predicted based on previously encoded images, a spatial prediction mode , according to which a block is predicted based on the previously coded blocks of the same image, interlayer prediction modes, according to which a block of an image showing the scene in a higher layer, as in a higher spatial resolution or from another point of view, is predicted based on a corresponding image that shows this scene in a lower layer, as in a lower spatial resolution or from another point of view.
[0108] A certain syntax is used to compile the quantized residual data 26, that is, levels of transformation coefficients and other residual data, as well as the data of the coding mode including, for example, the prediction modes and parameters of prediction for the individual blocks of the image 12 as determined by the predictor 18 and these syntax elements are subject to entropic encoding by entropic encoder 28. The data streams thus obtained as output by entropic encoder 28 are called byte sequence payload in gross 40.
[0109] The elements of the encoder 10 of Fig. 16 are interconnected, as can be seen in Fig. 16.
[0110] Fig. 17 shows a decoder that adapts to the encoder of Fig. 16, that is, that is capable of decoding the sequence payload of the raw bytes. The decoder of Fig. 17 is generally indicated by the reference signal 50 and comprises an entropic decoder 52, a retransforming / decanting module 54, a combiner 56, a filter 58 and a predictor 60. The entropic decoder 42 receives the payload of raw byte sequence 40 and performs entropic decoding using context adaptive binary arithmetic decoding to recover residual signal 62 and encoding parameters 64. The retransform / decantant modules 54 decant and retransform residual data 62 and forward the residual signal, thus obtained for combiner 56. Combiner 56 also receives a prediction signal 66 from predictor 60 which, in turn, forms prediction signal 66 using coding parameters 64 based on the reconstructed signal 68 determined by the combiner 56 when combining the prediction signal 66 and the residual signal 65. As already explained above in relation to Fig. 16, the predictor 60 can use the version It is not filtered from the reconstructed signal 68 or some intermediate version, alternatively or as a complement. The image to be reproduced and finally output at output 70 of decoder 50 can also be determined in an unfiltered version of the combination signal 68 or some filtered version thereof.
[0111] According to the tile concept, image 12 is subdivided into tiles and at least the block predictions within these tiles are restricted to the use, as the basis for a spatial prediction, of data merely related to the same tile. Through this measure, at least the prediction can be carried out in parallel for each tile individually. For illustrative purposes only, Fig. 16 illustrates image 12 as being subdivided into nine tiles. The subdivision of each tile in new blocks, as can be seen in Fig. 16, also serves merely as an example. In addition, to be complete, note that the way to code tiles separately may not be limited to spatial prediction (intra prediction). Instead, any prediction of coding parameters for a respective tile across the tile boundaries and any dependence on context selection in the entropic coding of a respective tile across the respective tile boundaries can also be prohibited, so as to be limited to be dependent only on data from the same tile. Therefore, the decoder is capable of carrying out the operations mentioned in parallel, namely in tile units.
[0112] To be transmitted through any transmission channel, the syntax elements must be entropically encoded piece by piece by entropic encoder 28. For this purpose, entropic encoder 28 analyzes the tile blocks, crossing the blocks of a first tile first and then proceed with the blocks of the next tile in order of tiles and so on. A scan analysis can, for example, be used to analyze the blocks within the tiles and the tiles, respectively. The pieces are then packaged in NAL units, which are the smallest units for transmission. Before entropically encoding a chunk, entropic encoder 28 initializes its CABAC probabilities, that is, the probabilities used to arithmetically encode the syntax element of that chunk. The entropic decoder 52 does the same, that is, it initializes its probabilities at the beginning of the chunk. Each initialization, however, negatively affects the efficiency of entropic coding, since the probabilities are continuously adapted to the current probability statistics, the symbol of the various contexts and the corresponding redefinition of the CABAC probabilities represents a deviation from the adapted state. As is known to any professional in the field, entropic coding leads to an optimized compression only if the probabilities adapt to the current probability statistics of the symbol.
[0113] Correspondingly, a decoder, according to a version of the present application, operates as shown in Fig. 18. The decoder receives in step 80 the payload of the raw byte sequence that describes an image 12 in tiles 82 , in tile plots. In Fig. 18, the first tile 82 in the order of tiles 84 is shown, by way of example, cut or divided into two portions 86a and 86b, each covering by way of example a sub-sequence of the block sequence within the tile. Then, in step 82, portions 86a and 86b are entropically decoded. However, in entropic decoding, plots 86a and 86b, the adaptation of the CABAC probabilities is continued beyond the plot limits. That is, during decoding of parcel 86a, CABAC probabilities are continuously adapted to the current symbol statistics and the state at the end of entropic decoding of parcel 86a is adapted when beginning entropic decoding of parcel 86b. In step 90, the payload of the raw byte sequence, thus entropically decoded, is decoded to obtain the image 12.
[0114] Due to the continuous adaptation of the CABAC probability beyond the 92 limits of the portion positioned on the interior tiles 82, these portion limits do not negatively affect the efficiency of entropic coding beyond the sub-division of the image 12 on tiles 82. On the other hand On the other hand, parallel processing of the tile is still possible. In addition, it is possible to transmit the parcels individually and, since the parcels are smaller than the complete tiles 82, it is possible to start at step 90 the decoding of each tile, as soon as the first parcel of the respective tile has been received and entropically decoded .
[0115] The description of Figs. 16 to 18 relate mainly to the use of tiles. As described above, tiles result from a spatial partition of an image. Similar to tiles, the pieces also spatially subdivide an image. The pieces are correspondingly also a means to allow parallel encoding / decoding. Similar to tiles, prediction and so on are prohibited, so that the pieces can be individually decoded. Correspondingly, the description of Figs. 16 to 18 is also valid for dividing pieces into plots.
[0116] The same applies when using WPP subflows. The WPP subflows also represent a spatial partition of an image 12, namely in WPP subflows. Unlike tiles and tiles, WPP subflows do not impose restrictions on predictions and contact selections through WPP subflows. The WPP subflows extend along the block rows, as LCU rows, as shown in Fig. 4, and to allow parallel processing, only a compromise is made with respect to the CABAC entropic coding in the order as defined from among the subflows WPP (see Fig. 4) 92 and for each WPP 92 subflow, except the first WPP subflow, the CABAC probabilities are not completely reset but are adopted or set to be equal, resulting in the CABAC probabilities after having entropically decoded the immediately preceding WPP subflow until the second LCU 94 from there, with the LCU order starting, for each WPP subflow, on the same side of the image 12, as the left side as shown in Fig. 4. Correspondingly, the same compliant coding delay between the sequence of WPP subflows, these WPP 92 subflows are decodable in parallel, so that the portions, to which the image 12 is decoded in parallel, that is, concurrently, forms a type of fr wave element 96 that moves across the image in a tiled way from left to right.
[0117] That is, when transferring the description of Fig. 16 to 18 for WPP subflows, any WPP 92 subflow (Fig. 4) can also be sub-divided into plots 98a and 98b without interrupting the adaptation of the CABAC probability at limit 100 between these portions 98a and 98b within the respective subflow WPP 92, thus avoiding penalties with respect to the efficiency of entropic coding due to the individual transmissibility of both portions 98a and 98b, but maintaining the ability to use parallel processing of front wave and allow the start of this parallel wavefront processing earlier, since the plots are smaller than the complete WPP 92 subflows.
[0118] As described above with reference to Figs. 1 through 15, there are several possibilities for transmitting parcels packaged in NAL units. Reference is made to Fig. 3, where the tiles or subflows or pieces of these plots or subflows were divided into plots in the arithmetically encoded domain with a header previous to that in the plot of each subflow or tile and present information that allows the location of the plot boundaries. . Another version was shown in Fig. 9.
[0119] There, the sub-division of tiles or WPP subflows into portions was made by the slight change in the structure of the piece: the pieces that start at a tile limit or WPP subflow, that is, that start at the beginning of a tile or WPP subflow, have the non_cabac_reset_bandeira set to zero, thus causing the usual initialization / reset of the CABAC probability. The pieces, however, that carry parcels that start inside a tile or WPP subflow have no_cabac_reset_bandeira set to one, thus causing the continuation described above of the adaptation of the CABAC probability.
[0120] With regard to deinterleaving, which takes place in reception step 80, for each parcel it is determined to which subflow WPP or tile belongs to the respective parcel. The different possibilities have been described above, for example, a round-robin cycle through the number of WPP subflows or tiles in a current image. Alternatively, in the case of using chunk headers to transport the parcels, the chunk headers can comprise an indication that allows the location of the beginning of the respective chunk within the current image 12.
[0121] In this regard, note that the decomposition of pieces, WPP subflows or tiles into plots is carried out along an order, decoding within each piece, WPP subflow or tile: that is, within each piece, WPP subflow or tile, the portion of the image spatially covered by the respective piece, WPP subflow or tile, is encoded in, or decoded from, the respective piece, WPP subflow or tile in that decoding order, and each portion of a respective piece, WPP subflow or tile covers a continuous portion of the respective piece, WPP subflow or tile along the decoding order. In this way, an order is defined among parcels that belong to the same piece, WPP subflow or tile, namely the encoding / decoding order, and each parcel has a classification within that order. Once the subdivision of the image into WPP subflows or tiles is signaled to the decoder, the decoder is informed about the subdivisions. Correspondingly, to associate each parcel with a respective WPP or tile subflow, for example, it is sufficient if each parcel has a start address that identifies a starting position, from which the respective parcel continuously covers the image using the order of coding / decoding for the WPP tile / subflows, of which the respective portion is part. Even the order of the parcels that belong to a certain tile or WPP subflow, for example, can be reconstructed in a transport demultiplexer or by the decoder using the initial positions. However, to reclassify, information from the transport packet headers of the lower OSI layers can also be used, as described above for RTP transmissions, such as the decoding order number, i.e. DONs. A transport demultiplexer of the mentioned type can be configured in the same way as the transport demultiplexer discussed above, to store data from WPP subflow plots or the same tile in a temporary piece memory, and the data from WPP subflow plots or tiles associated different WPP subflows or tiles in different pieces' temporary memories. As mentioned above, the chunk structure, that is, the chunk headers, can be used to transport parcels.
[0122] Next, reference is made to the versions of Figs. 11 to 15 to describe them again in other words. As described in these figures, the Si pieces are packaged in NAL units with each NAL 110 unit (see Fig. 11) comprising a header from the NAL 112 unit. Note that the Si pieces can be normal pieces or pieces that carry parcels according to Fig. 9. Correspondingly, these pieces only carry data that pertains to a WPP subflow or tile of a current image, namely the WPP or tile subflow, respectively. Through fragmentation, NAL units 110 are transported via transport flow packets (TS) 114, namely the payload section 116 thereof. In doing so, each NAL 110 unit and the corresponding Si piece is preceded by a respective MKR subflow marker indicating i, that is, the WPP or tile subflow, to which the piece immediately after the NAL 110 unit belongs immediately to follow.
[0123] NAL 110 units, which carry pieces that belong to different WPP subflows or tiles, can be distributed in more than one elementary flow ES or to the same elementary flow, as explained in Figs. 11 through 13. As mentioned above, the “elementary stream” can also identify a separate RTP stream in its own RTP session.
[0124] As explained in relation to Figs. 14 and 15, a transport demultiplexer may comprise a multiplex MB buffer, SB chip buffers and a TB transport buffer. The temporary memories of SB pieces are pulled by a decoder and multiple chaining that allows a parallel decoding of an image in subflows or WPP tiles; The TB transport buffer configured to collect data belonging to a TS packet from a predetermined elementary stream of a video bit stream, and forward the data to the MB buffer. The transport demultiplexer is then configured to evaluate NAL unit headers from NAL units from a sequence of NAL units packaged in TS packets in an output of the MB buffer, discard MKR of NAL units from subflow markers with storage of marker data. subflow carried within the NAL units of subflow markers and store data from pieces of subflows or tiles within NAL units that follow NAL units of subflow markers, whose data field identifies a WPP subflow or paca equal to one, that is, the same, temporary memory of SB pieces, and data of pieces of subflows or WPP tiles within NAL units that follow NAL units of subflow markers, whose data field identifies different subflows or WPP tiles in different temporary memories of SB pieces. As can be seen in Fig. 15, the transport demultiplexer can comprise a demultiplexer called TS demux in Fig. 15, and is configured to receive the video bit stream and split TS packets of the video bit stream into different elementary streams. , that is, to distribute the TS packets of the video byte stream to different elementary streams. The demultiplexer performs this division or distribution according to the PIDs contained in the TS headers of the TS packet, so that each elementary flow is composed of TS packets from a PAD of TS packets from other elementary flows.
[0125] That is, if the pieces correspond to the parcels in the sense of the version of Fig. 9, the MTD, that is, the multi-thread decoder is capable of initiating the processing of more than one WPP subflow or tile of an image current, as soon as the corresponding SB piece buffer of the respective WPP or tile subflow has data contained, thus reducing the delay.
[0126] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, in which a block or device corresponds to a method step or to a characteristic of a method step . Similarly, the aspects described in the context of a method step also represent a description of a corresponding block or item or characteristic of a corresponding apparatus. Some or all of the method steps can be performed (or used) by a hardware device, such as a microprocessor, a programmable computer or an electronic circuit. In some versions, one or more of the most important steps in the method can be performed by such a device.
[0127] The encoded bit stream of the invention can be stored on a digital storage medium or it can be transmitted on a transmission medium, such as a wireless transmission medium or a wired transmission medium, such as the Internet.
[0128] These contributions above thus describe, among other things, methods for low-delay encapsulation and transmission of structured video data as provided by the new HEVC encoding standard, as structured in tiles, wavefront parallel processing subflows ( WPP), entropic pieces or pieces. Among other things, techniques were presented that allow low delay transport in a parallelized encoder - transmitter - receiver - decoder environment through the interleaved transport of entropic pieces / pieces / tiles / subflows. To solve the bottleneck problems outlined in the introductory part of the specification and to minimize the delay in transmission and decoding time, that is, the end-to-end delay, a technique was introduced for an interleaved piece scheme for parallel transmission and processing .
[0129] Depending on certain implementation requirements, versions of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, with read control signals electronics stored there, that cooperate (or are able to cooperate) with a programmable computer system, so that the respective method is executed. Therefore, the digital storage medium can be read on a computer.
[0130] Some versions according to the invention comprise a data carrier with electronic reading control signals, which are able to cooperate with a programmable computer system, so that one of the methods described here is performed.
[0131] Generally speaking, the versions of the present invention can be implemented as a computer program product with a program code, the program code being operative to execute one of the methods when the computer program product runs on a computer. The program code can, for example, be stored on a machine-readable medium.
[0132] Other versions include the computer program to execute one of the methods described here, stored on a machine-readable medium.
[0133] In other words, a version of the method of the invention is, therefore, a computer program with a program code to execute one of the methods described here, when the computer program runs on a computer.
[0134] Another version of the methods of the invention is, therefore, a data medium (or a digital storage medium, or a computer reading medium) comprising, recorded there, the computer program to execute one of the methods described here . The data medium, the digital storage medium or the recorded medium are typically tangible and / or non-transitional.
[0135] Another version of the method of the invention is, therefore, a data stream or a sequence of signals representing the computer program to execute one of the methods described here.
[0136] The data stream or the signal sequence can, for example, be configured to be transferred via a data communication link, for example via the Internet.
[0137] Another version comprises a processing medium, for example, a computer, or a programmable logic device, configured for or adapted to execute one of the methods described here.
[0138] Another version comprises a computer with the computer program installed to execute one of the methods described here.
[0139] Another version according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program to perform one of the methods described herein to a receiver. The receiver can, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
[0140] In some versions, a programmable logic device (for example, a network of programmable logic gates) can perform some or all of the functionality of the methods described here. In some versions, a network of programmable logic gates can cooperate with a microprocessor. to perform one of the methods described here. In general, the methods are preferably performed by any hardware device.
[0141] The versions described above are merely illustrated for the principles of the present invention. It is understood that the changes and variations in the provisions and details described will be evident to professionals in the field. It is, therefore, intended to be limited only by the scope of the pending patent claims and not by the specific details of the description and explanation of the versions contained herein. REFERENCES
[0142] Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, Ajay Luthra, ”Overview of the H.264 / AVC Video Coding Standard”, IEEE Trans. System. Tecnol Circuits. Video, vol. 13, N7, July 2003.
[0143] JCTVC-E196, “Wavefront Parallel Processing”, 5th JCT-VC Meeting, Geneva 2011.
[0144] JCTVC-D070, “Lightweight slicing for entropy coding”, 4th Meeting, Daegu, 2011.
[0145] JCTVC-D073, “Periodic initialization for wavefront coding functionality”, 4th Meeting, Daegu, 2011.
[0146] HEVC WD5: Draft 5 of the High Efficiency of Video Coding JTCVC-G1103, 5th JCT-VC Meeting, Geneva Meeting of November 2011.
[0147] JTCVC-D243, "Analysis of entropy slices approaches", 4th Meeting, Daegu, 2011.
[0148] ISO / IEC 13818-1 / 2011, MPEG-2 Transport Flow including AMDs 1 - 6.
[0149] IETF Real-time transport protocol, RTP RFC 3550.
[0150] IETF RTP Payload Format, IETF RFC 6184.
[0151] JCTVC-F275, Wavefront and Cabac Flow: Different Degrees of Parallelism Without Transcoding, Torino Meeting
[0152] JCT-VC-F724, Wavefront Parallel Processing for HEVC Encoding and Decoding, Torino Meeting ** at the end of the description
[0153] IETF Session Description Protocol (SDP), RFC 4566
[0154] IETF RTP Payload Format for High Efficiency of Video Encoding, draft-schierl-payload-h265.
权利要求:
Claims (10)
[0001]
1. Decoder configured to decode the data representing an encoded figure in a video, characterized in that the decoder comprises: an entropy decoder configured to decode the data representing the encoded figure using the entropy decoding of the context-adaptive binary arithmetic encoding ( CABAC) to obtain a residual signal associated with the figure, in which each line of the figure is encoded as a continuous wavefront parallel processing (WPP) subflow, in which at least one WPP continuous subflow of the figure includes a plurality of plots, where each plot includes information that indicates its delimitation in at least one continuous WPP subflow, where the entropy decoder is configured to decode in entropy the plurality of plots using CABAC entropy decoding with continuous CABAC probability adaptation to from a first of the plurality of plots in at least one subflow continuous WPP by delimiting the first portion in relation to a second of the plurality of parcels in at least one continuous WPP subflow, and a combiner to combine the residual signal and a forecast signal to reconstruct the figure.
[0002]
2. Decoder, according to claim 1, characterized in that the parcels are packaged using parcel headers, and the entropy decoder is configured so that, upon receipt of the plurality of parcels, it is responsive, upon receipt of a new parcel, the an indicator in the header of the parcel indicating the new parcel, a slice_type of the new parcel or a type of the NAL unit of an NAL unit comprising the new parcel, in such a way that the entropy decoder is configured to, based on the indicator, both interrupt the CABAC probability adaptation by redefining the CABAC probabilities and to continue the CABAC probability adaptation.
[0003]
3. Decoder, according to claim 1, characterized in that the decoder is configured to deinterleave the plurality of portions of the WPP continuous subflows by identifying, for each portion, a continuous WPP subflow to which the respective portion belongs.
[0004]
4. Decoder according to claim 1, characterized in that the parcels are packaged in packages in such a way that each package comprises a portion of each WPP continuous subflow of the figure, or a subset of the continuous WPP subflows of the figure, arranged in an order defined between the WPP continuous subflows, each package comprising a header that comprises the information that reveals the positions and / or the lengths of the parcels packaged in the respective package, or markers that separate the parcels in the respective package from each other, where the decoder is configured to use the information understood by the headers or the markers to access the parcels in the packages.
[0005]
5. Decoder, according to claim 4, characterized in that the packages that comprise first, according to the order defined between the WPP continuous subflows, portions of the WPP continuous subflows of the figure, comprise a low delay resource indicator, and the packets comprising second or subsequent, according to the order defined between the WPP continuous subflows, portions of the WPP continuous subflows of the figure, comprise a continuation indicator.
[0006]
6. Decoder, according to claim 4, characterized in that the packages are NAL units.
[0007]
7. Encoder configured to encode a picture from a video, characterized in that the encoder comprises: a residual determiner configured to determine a residual signal associated with the figure based on a prediction signal, and an entropy encoder configured to encode the residual signal associated with the figure in a continuous stream of data using the entropy coding CABAC, where each line of the figure is encoded as a WPP continuous subflow, at least one WPP continuous subflow of the figure including a plurality of plots, where each plot includes information that indicates a delimitation of the same in at least one continuous WPP subflow, in which the entropy encoder is configured to entropy the plurality of portions by using the CABAC entropy coding with continuous CABAC probability adaptation from a first of the plurality of plots in at least one continuous WPP subflow by delimiting the first plot in with respect to a second of the plurality of plots in at least one continuous WPP subflow.
[0008]
8. Encoder according to claim 7, characterized in that the encoder is configured to form the plurality of portions in such a way that each portion is of a maximum transfer unit size.
[0009]
9. Method for decoding data representing a coded picture from a video, characterized in that the method comprises: receiving the data representing the coded picture, in which each line of the picture is coded as a WPP continuous subflow, at least one continuous subflow Figure WPP including a plurality of plots, where each plot includes information indicating its delimitation in at least one WPP, entropy decoding the data representing the encoded figure using CABAC entropy decoding to obtain a residual signal associated with the figure, entropy decoding comprising entropy decoding the plurality of plots using CABAC entropy decoding with continuous CABAC probability adaptation from a first of the plurality of plots in at least one WPP continuous subflow by delimiting the first plot with respect to a second of the plurality of plots in at least one continuous WPP subflow, and the figure by the combination of the residual signal and a forecast signal.
[0010]
10. Method for encoding a figure from a video, characterized in that the method comprises: determining a residual signal associated with the figure based on a forecast signal, and entropy encoding the residual signal associated with the figure in a continuous flow of data using CABAC entropy coding, where each line of the figure is encoded as a WPP continuous subflow, at least one WPP continuous subflow of the figure including a plurality of plots, where each plot includes information indicating a delimitation of the same in the hair at least one WPP continuous subflow, where entropy encoding comprises entropy encoding the plurality of plots by using the CABAC entropy encoding with continuous CABAC probability adaptation from a first of the plurality of plots in at least one WPP continuous subflow through the delimitation of the first plot in relation to a second of the plurality of plots in at least one continuous WPP subflow.
类似技术:
公开号 | 公开日 | 专利标题
BR112014017915B1|2021-03-16|coding concept that allows parallel processing, transport demultiplexer and video byte stream
BR122020007529B1|2021-09-21|CODING CONCEPT ALLOWING PARALLEL PROCESSING, TRANSPORT DEMULTIPLEXER AND VIDEO BITE FLOW
TWI753214B|2022-01-21|Encoder/decoder allowing parallel processing, transport demultiplexer, system, storage medium, method and computer program
同族专利:
公开号 | 公开日
KR20150003407A|2015-01-08|
CN109729357A|2019-05-07|
CN109729350A|2019-05-07|
BR112014017915A2|2019-10-08|
KR20170078852A|2017-07-07|
RU2019102609A|2020-07-30|
AU2016202208A1|2016-05-05|
US20140334557A1|2014-11-13|
JP6721638B2|2020-07-15|
AU2013211002B2|2016-01-14|
KR20170078850A|2017-07-07|
CA2861951C|2020-08-11|
US10880577B2|2020-12-29|
KR20200010606A|2020-01-30|
EP3923578A1|2021-12-15|
RU2019102609A3|2021-11-23|
PH12018500345B1|2018-07-09|
US20180167641A1|2018-06-14|
KR101862329B1|2018-05-29|
KR101863177B1|2018-05-31|
KR20140131926A|2014-11-14|
EP2805491B1|2021-05-12|
TW201728174A|2017-08-01|
JP2020167719A|2020-10-08|
US20180167640A1|2018-06-14|
JP6808341B2|2021-01-06|
WO2013107906A3|2013-09-26|
AU2020244429A1|2020-10-29|
RU2679551C2|2019-02-11|
IL233699D0|2014-09-30|
JP2018198447A|2018-12-13|
ES2886119T3|2021-12-16|
KR20180059571A|2018-06-04|
CA2861951A1|2013-07-25|
PH12018500347B1|2018-07-09|
KR20170078851A|2017-07-07|
IL267776A|2021-07-29|
RU2610291C2|2017-02-08|
IL251266D0|2017-05-29|
MY167465A|2018-08-29|
HK1204182A1|2015-11-06|
IN2014KN01699A|2015-10-23|
KR101752879B1|2017-06-30|
CN109729356A|2019-05-07|
US10880579B2|2020-12-29|
PH12018500346B1|2018-07-09|
US20190124365A1|2019-04-25|
RU2014134041A|2016-03-20|
CL2014001886A1|2014-11-21|
US20190306537A1|2019-10-03|
CN104081781A|2014-10-01|
RU2017103637A|2019-01-18|
PH12018500344B1|2018-07-09|
CN104081781B|2018-11-02|
PH12018500346A1|2018-07-09|
CN109729355A|2019-05-07|
JP2016158267A|2016-09-01|
KR20170077295A|2017-07-05|
PH12018500344A1|2018-07-09|
UA114618C2|2017-07-10|
US10873766B2|2020-12-22|
IL251266A|2018-10-31|
AU2020244429B2|2021-09-30|
PH12014501660B1|2014-10-13|
PH12018500345A1|2018-07-09|
KR102070887B1|2020-01-29|
KR101718488B1|2017-03-21|
IL284595D0|2021-08-31|
US9930368B2|2018-03-27|
SG10201606621XA|2016-09-29|
IL233699A|2017-04-30|
KR101863181B1|2018-05-31|
MX2014008695A|2014-11-21|
PH12018500347A1|2018-07-09|
IL262644D0|2018-12-31|
AU2019226180B2|2020-07-02|
IL284595A|2022-03-01|
PH12014501660A1|2014-10-13|
US10880578B2|2020-12-29|
AU2016202208B2|2017-05-04|
JP2015507899A|2015-03-12|
AU2017210565B2|2019-06-20|
KR101863179B1|2018-05-31|
ZA201406040B|2015-12-23|
CA3081964A1|2013-07-25|
SG11201404251QA|2014-08-28|
AU2021290315A1|2022-01-27|
US20190124364A1|2019-04-25|
MX350507B|2017-09-08|
TW201921939A|2019-06-01|
TWI645715B|2018-12-21|
RU2017103637A3|2019-01-18|
TWI559744B|2016-11-21|
AU2019226180A1|2019-09-26|
IL267776D0|2019-08-29|
TW201336314A|2013-09-01|
WO2013107906A2|2013-07-25|
IL262644A|2019-07-31|
UA124569C2|2021-10-13|
US20210044835A1|2021-02-11|
EP3923578A4|2021-12-15|
SG10202001107WA|2020-04-29|
US10887625B2|2021-01-05|
KR102210228B1|2021-02-01|
AU2017210565A1|2017-08-24|
EP2805491A2|2014-11-26|
AU2013211002A1|2014-09-18|
KR20210012066A|2021-02-02|
引用文献:
公开号 | 申请日 | 公开日 | 申请人 | 专利标题

US5926205A|1994-10-19|1999-07-20|Imedia Corporation|Method and apparatus for encoding and formatting data representing a video program to provide multiple overlapping presentations of the video program|
US6683909B1|2000-03-16|2004-01-27|Ezenial Inc.|Macroblock parsing without processing overhead|
US7970058B2|2002-07-15|2011-06-28|Hitachi Consumer Electronics Co., Ltd.|Moving picture encoding method and decoding method|
EP1582021A4|2003-01-09|2009-04-08|Thomson Licensing|A method and an apparatus for mapping an mpeg transport stream into ip packets for wlan broadcast|
US7688895B2|2003-07-22|2010-03-30|Lsi Corporation|Method and/or circuit for binary arithmetic decoding decisions before termination|
US7558954B2|2003-10-31|2009-07-07|Hewlett-Packard Development Company, L.P.|Method and apparatus for ensuring the integrity of data|
JP4931034B2|2004-06-10|2012-05-16|株式会社ソニー・コンピュータエンタテインメント|Decoding device, decoding method, program, and program recording medium|
US20060002479A1|2004-06-22|2006-01-05|Fernandes Felix C A|Decoder for H.264/AVC video|
US20060062312A1|2004-09-22|2006-03-23|Yen-Chi Lee|Video demultiplexer and decoder with efficient data recovery|
US7675872B2|2004-11-30|2010-03-09|Broadcom Corporation|System, method, and apparatus for displaying pictures|
CA2633637C|2006-01-05|2014-06-17|Nippon Telegraph And Telephone Corporation|Video encoding method and decoding method, apparatuses therefor, programs therefor, and storage media for storing the programs|
US8315308B2|2006-01-11|2012-11-20|Qualcomm Incorporated|Video coding with fine granularity spatial scalability|
JP2007214998A|2006-02-10|2007-08-23|Fuji Xerox Co Ltd|Coding apparatus, decoding apparatus, coding method, decoding method, and program|
EP2039168A2|2006-07-05|2009-03-25|Thomson Licensing|Methods and apparatus for multi-view video encoding and decoding|
US20080040498A1|2006-08-10|2008-02-14|Nokia Corporation|System and method of XML based content fragmentation for rich media streaming|
JP2008141483A|2006-12-01|2008-06-19|Brother Ind Ltd|Tree type distribution system, node device, information processing program, and information distribution method|
KR100849495B1|2006-12-04|2008-07-31|한국전자통신연구원|Method for producing bit-rate based on RTP packet mode|
CN101198051B|2006-12-07|2011-10-05|深圳艾科创新微电子有限公司|Method and device for implementing entropy decoder based on H.264|
CA2668003C|2007-08-22|2013-04-02|Nippon Telegraph And Telephone Corporation|Video quality estimation device, video quality estimation method, frame type judgment method, and recording medium|
US8938009B2|2007-10-12|2015-01-20|Qualcomm Incorporated|Layered encoded bitstream structure|
CN101170688B|2007-11-26|2010-12-01|电子科技大学|A quick selection method for macro block mode|
US8542748B2|2008-03-28|2013-09-24|Sharp Laboratories Of America, Inc.|Methods and systems for parallel video encoding and decoding|
JP4875024B2|2008-05-09|2012-02-15|株式会社東芝|Image information transmission device|
US8619856B2|2008-10-03|2013-12-31|Qualcomm Incorporated|Video coding with large macroblocks|
US7932843B2|2008-10-17|2011-04-26|Texas Instruments Incorporated|Parallel CABAC decoding for video decompression|
US9467699B2|2008-12-03|2016-10-11|Hfi Innovation Inc.|Method for performing parallel coding with ordered entropy slices, and associated apparatus|
KR101063424B1|2009-02-02|2011-09-07|주식회사 코아로직|Video data processing device and method|
JP2010278668A|2009-05-27|2010-12-09|Sony Corp|Coding apparatus and method, and decoding apparatus and method|
JP5212319B2|2009-09-08|2013-06-19|ブラザー工業株式会社|Encoding apparatus, encoding method, and encoding program|
US20110194613A1|2010-02-11|2011-08-11|Qualcomm Incorporated|Video coding with large macroblocks|
TWI403170B|2010-05-21|2013-07-21|Univ Nat Chiao Tung|Context-based adaptive binary arithmetic coding device and coding method thereof|
US8344917B2|2010-09-30|2013-01-01|Sharp Laboratories Of America, Inc.|Methods and systems for context initialization in video coding and decoding|
US20120163457A1|2010-12-28|2012-06-28|Viktor Wahadaniah|Moving picture decoding method, moving picture coding method, moving picture decoding apparatus, moving picture coding apparatus, and moving picture coding and decoding apparatus|
US9370647B2|2011-07-14|2016-06-21|W. L. Gore & Associates, Inc.|Expandable medical devices|
KR101882979B1|2011-07-15|2018-08-24|지이 비디오 컴프레션, 엘엘씨|Sample array coding for low-delay|
KR101863181B1|2012-01-20|2018-05-31|지이 비디오 컴프레션, 엘엘씨|Coding concept allowing parallel processing, transport demultiplexer and video bitstream|
PL2842313T3|2012-04-13|2017-06-30|Ge Video Compression, Llc|Scalable data stream and network entity|FR2972588A1|2011-03-07|2012-09-14|France Telecom|METHOD FOR ENCODING AND DECODING IMAGES, CORRESPONDING ENCODING AND DECODING DEVICE AND COMPUTER PROGRAMS|
FR2977111A1|2011-06-24|2012-12-28|France Telecom|METHOD FOR ENCODING AND DECODING IMAGES, CORRESPONDING ENCODING AND DECODING DEVICE AND COMPUTER PROGRAMS|
KR101863181B1|2012-01-20|2018-05-31|지이 비디오 컴프레션, 엘엘씨|Coding concept allowing parallel processing, transport demultiplexer and video bitstream|
KR102312989B1|2012-03-22|2021-10-14|엘지전자 주식회사|Video encoding method, video decoding method and apparatus using same|
CN104221388B|2012-04-16|2018-01-23|瑞典爱立信有限公司|Indicate the fixation slice structure mark of the parallel processing possibility of compression of video sequence|
JP6080405B2|2012-06-29|2017-02-15|キヤノン株式会社|Image encoding device, image encoding method and program, image decoding device, image decoding method and program|
US9621905B2|2012-06-29|2017-04-11|Qualcomm Incorporated|Tiles and wavefront parallel processing|
WO2014024491A1|2012-08-09|2014-02-13|パナソニック株式会社|Image decoding method, image encoding method, image decoding device, image encoding device, and image encoding/decoding device|
US9491457B2|2012-09-28|2016-11-08|Qualcomm Incorporated|Signaling of regions of interest and gradual decoding refresh in video coding|
ES2733223T3|2013-01-04|2019-11-28|Samsung Electronics Co Ltd|Entropy decoding procedure of cutting segments|
US9924165B1|2013-07-03|2018-03-20|Ambarella, Inc.|Interleaved video coding pipeline|
US9628792B2|2013-07-15|2017-04-18|Qualcomm Incorporated|Cross-layer parallel processing and offset delay parameters for video coding|
CN111510725A|2013-07-15|2020-08-07|Ge视频压缩有限责任公司|Network device and error handling|
JP6268066B2|2013-09-20|2018-01-24|パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America|Transmission method, reception method, transmission device, and reception device|
US9305325B2|2013-09-25|2016-04-05|Apple Inc.|Neighbor context caching in block processing pipelines|
US9299122B2|2013-09-25|2016-03-29|Apple Inc.|Neighbor context processing in block processing pipelines|
US9270999B2|2013-09-25|2016-02-23|Apple Inc.|Delayed chroma processing in block processing pipelines|
US9596470B1|2013-09-27|2017-03-14|Ambarella, Inc.|Tree-coded video compression with coupled pipelines|
US9215472B2|2013-09-27|2015-12-15|Apple Inc.|Parallel hardware and software block processing pipelines|
US9218639B2|2013-09-27|2015-12-22|Apple Inc.|Processing order in block processing pipelines|
US9571846B2|2013-09-27|2017-02-14|Apple Inc.|Data storage and access in block processing pipelines|
US9578372B2|2013-09-27|2017-02-21|Cisco Technology, Inc.|Delay tolerant decoder|
WO2015054813A1|2013-10-14|2015-04-23|Microsoft Technology Licensing, Llc|Encoder-side options for intra block copy prediction mode for video and image coding|
RU2666635C2|2013-10-14|2018-09-11|МАЙКРОСОФТ ТЕКНОЛОДЖИ ЛАЙСЕНСИНГ, ЭлЭлСи|Features of base colour index map mode for video and image coding and decoding|
CN105765974B|2013-10-14|2019-07-02|微软技术许可有限责任公司|Feature for the intra block of video and image coding and decoding duplication prediction mode|
US10390034B2|2014-01-03|2019-08-20|Microsoft Technology Licensing, Llc|Innovations in block vector prediction and estimation of reconstructed sample values within an overlap area|
EP3090553A4|2014-01-03|2017-12-20|Microsoft Technology Licensing, LLC|Block vector prediction in video and image coding/decoding|
KR101847899B1|2014-02-12|2018-04-12|주식회사 칩스앤미디어|Method and apparatus for processing video|
US10542274B2|2014-02-21|2020-01-21|Microsoft Technology Licensing, Llc|Dictionary encoding and decoding of screen content|
US10368091B2|2014-03-04|2019-07-30|Microsoft Technology Licensing, Llc|Block flipping and skip mode in intra block copy prediction|
CA2950171C|2014-05-28|2019-11-19|Arris Enterprises Llc|Content aware scheduling in a hevc decoder operating on a multi-core processor platform|
WO2015192353A1|2014-06-19|2015-12-23|Microsoft Technology Licensing, Llc|Unified intra block copy and inter prediction modes|
US9807410B2|2014-07-02|2017-10-31|Apple Inc.|Late-stage mode conversions in pipelined video encoders|
CN105874795B|2014-09-30|2019-11-29|微软技术许可有限责任公司|When wavefront parallel processing is activated to the rule of intra-picture prediction mode|
US9591325B2|2015-01-27|2017-03-07|Microsoft Technology Licensing, Llc|Special case handling for merged chroma blocks in intra block copy prediction mode|
US10027989B2|2015-05-06|2018-07-17|Integrated Device Technology, Inc.|Method and apparatus for parallel decoding|
CN106664405B|2015-06-09|2020-06-09|微软技术许可有限责任公司|Robust encoding/decoding of escape-coded pixels with palette mode|
US10003811B2|2015-09-01|2018-06-19|Microsoft Technology Licensing, Llc|Parallel processing of a video frame|
US20170105010A1|2015-10-09|2017-04-13|Microsoft Technology Licensing, Llc|Receiver-side modifications for reduced video latency|
US10230948B2|2016-02-03|2019-03-12|Mediatek Inc.|Video transmitting system with on-the-fly encoding and on-the-fly delivering and associated video receiving system|
CN109076144B|2016-05-10|2021-05-11|奥林巴斯株式会社|Image processing apparatus, image processing method, and storage medium|
CN107483950A|2016-06-07|2017-12-15|北京大学|Picture parallel encoding method and system|
US20180020228A1|2016-07-12|2018-01-18|Mediatek Inc.|Video processing system with multiple syntax parsing circuits and/or multiple post decoding circuits|
WO2018207956A1|2017-05-10|2018-11-15|엘지전자|Method and device for entropy-encoding and entropy-decoding video signal|
GB2569269B|2017-10-20|2020-07-15|Graphcore Ltd|Synchronization in a multi-tile processing arrangement|
GB2569645B|2017-12-22|2022-02-23|Displaylink Uk Ltd|Managing data for transportation|
US10986349B2|2017-12-29|2021-04-20|Microsoft Technology Licensing, Llc|Constraints on locations of reference blocks for intra block copy prediction|
WO2020003266A1|2018-06-29|2020-01-02|Beijing Bytedance Network Technology Co., Ltd.|Resetting of look up table per slice/tile/lcu row|
CN110662064A|2018-06-29|2020-01-07|北京字节跳动网络技术有限公司|Checking order of motion candidates in LUT|
WO2020003282A1|2018-06-29|2020-01-02|Beijing Bytedance Network Technology Co., Ltd.|Managing motion vector predictors for video coding|
CN114125450A|2018-06-29|2022-03-01|北京字节跳动网络技术有限公司|Method, apparatus and computer readable medium for processing video data|
EP3794824A1|2018-06-29|2021-03-24|Beijing Bytedance Network Technology Co. Ltd.|Conditions for updating luts|
JP2021530936A|2018-06-29|2021-11-11|北京字節跳動網絡技術有限公司Beijing Bytedance Network Technology Co., Ltd.|Look-up table updates: FIFO, restricted FIFO|
GB2588528A|2018-06-29|2021-04-28|Beijing Bytedance Network Tech Co Ltd|Selection of coded motion information for LUT updating|
TWI735902B|2018-07-02|2021-08-11|大陸商北京字節跳動網絡技術有限公司|Lookup table with intra frame prediction and intra frame predication from non adjacent blocks|
TW202025760A|2018-09-12|2020-07-01|大陸商北京字節跳動網絡技術有限公司|How many hmvp candidates to be checked|
CN113170237A|2018-09-28|2021-07-23|中兴通讯股份有限公司|Video encoding and decoding method and apparatus|
US11206244B2|2018-12-21|2021-12-21|ARRIS Enterprise LLC|Method to preserve video data obfuscation for video frames|
CN113383554A|2019-01-13|2021-09-10|北京字节跳动网络技术有限公司|Interaction between LUTs and shared Merge lists|
WO2020197236A1|2019-03-24|2020-10-01|엘지전자 주식회사|Image or video coding based on sub-picture handling structure|
WO2020231220A1|2019-05-15|2020-11-19|현대자동차주식회사|Method and apparatus for parallel encoding and decoding of moving picture data|
US20210105022A1|2019-10-02|2021-04-08|Apple Inc.|Low-Latency Encoding Using a Bypass Sub-Stream and an Entropy Encoded Sub-Stream|
WO2021252978A1|2020-06-12|2021-12-16|Bytedance Inc.|Constraints on picture output ordering in a video bitstream|
CN111726626A|2020-06-18|2020-09-29|上海兆芯集成电路有限公司|Integrated circuit and probability table storage method for video decoding|
CN112995237B|2021-05-21|2021-10-08|杭州博雅鸿图视频技术有限公司|Method, device and equipment for processing video data stream and storage medium|
法律状态:
2019-10-22| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|
2020-02-27| B25A| Requested transfer of rights approved|Owner name: GE VIDEO COMPRESSION, LLC (US) |
2020-09-08| B06A| Patent application procedure suspended [chapter 6.1 patent gazette]|
2021-01-05| B15K| Others concerning applications: alteration of classification|Free format text: A CLASSIFICACAO ANTERIOR ERA: H04N 7/26 Ipc: H04N 19/13 (2014.01), H04N 19/436 (2014.01), H04N |
2021-01-12| B09A| Decision: intention to grant [chapter 9.1 patent gazette]|
2021-03-16| B16A| Patent or certificate of addition of invention granted [chapter 16.1 patent gazette]|Free format text: PRAZO DE VALIDADE: 20 (VINTE) ANOS CONTADOS A PARTIR DE 21/01/2013, OBSERVADAS AS CONDICOES LEGAIS. |
优先权:
申请号 | 申请日 | 专利标题
US201261588849P| true| 2012-01-20|2012-01-20|
US61/588,849|2012-01-20|
PCT/EP2013/051043|WO2013107906A2|2012-01-20|2013-01-21|Coding concept allowing parallel processing, transport demultiplexer and video bitstream|
[返回顶部]