• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    method and device for providing supplementary processing information relating to encoded media content, method for processing a media content, and, media terminal. supplementary processing information (52, 54) defined in post-decoding instructions for processing decoded media content is co-organized into a file (1) together with a pseudo-identifier of a media codec employed to encode the media content. the trigger abort of media content decoding co-organization by a legacy media terminal (210) that does not recognize the pseudo-identifier, but enables decoding of the media content by a media terminal (200) that recognizes the pseudo-identifier hereby to enable post-decoding processing of the decoded media content using the supplementary processing information (52,54). co-organizing by this means prevents legacy media terminals (210) that cannot use the supplementary processing information from decoding the media content and representing the media content in a media presentation.
    公开号:BR112012009148B1
    申请号:R112012009148-4
    申请日:2010-10-18
    公开日:2021-05-25
    发明作者:Per Fröjdh；Clinton Priddle；Zhuangfei Wu
    申请人:Telefonaktiebolaget Lm Ericsson (Publ)；
    IPC主号:
    专利说明:

    TECHNICAL FIELD
    Embodiments generally relate to media content processing, and in particular to providing useful supplementary processing information regarding media content processing. FOUNDATION
    H.264/MPEG-4 Part 1O or Advanced Video Coding (AVC) is the state of the art video coding standard. It is a hybrid codec that takes advantage of eliminating redundancy between frames and within a frame. The output of the encoding process is Video Coding Layer (VCL) data which is further encapsulated in Network Abstraction Layer (NAL) unit prior to transmission or storage.
    Supplemental Enhancement Information (SEI) is provided to assist with post-decoding related processes such as display. Examples of SEI messages include defining post-filter suggestions and tone mapping. SEI messages are designed in such a way that they are not mandatory for the decoding process. Conformation decoders can drop SEI messages they don't understand. SEI messages are now encapsulated in a NAL unit just like VCL data, but do not affect decoding.
    According to the prior art, a file must be made with specific knowledge of the display capabilities of media terminals to ensure correct playback. New display features plus enhancement information, ie, outside the decoding mesh, cannot be made in a backward compatible way. An example of this is the spatially interleaved frame SEI messages that have been added to H.264 video to help represent decoded images on three-dimensional (3D) display devices. These SEI messages are derived from the effort to use an existing two-dimensional (2D) codec, such as H.264, to encode 3D content. An example of this is where a SEI message describes that even frames are allocated to the left view, odd frames are allocated to the right view.
    SEI messages are designed in a throwaway way when it comes to decoding. It is important, however, that the media terminal can correctly interpret this information for things outside of decoding, such as 3D representation. A legacy media terminal that does not have the ability to understand these SEI messages would simply ignore them and represent the decoded images in a 2D display resulting in significant display artifacts, for example jumping between left and right views at every frame, to the example shown above.
    Furthermore, today a media terminal needs to decode the video bitstream in order to find out if and what types of enhancement information are needed. It is problematic for both playback and local broadcast. For example, someone may have a movie that is not fully viewable and the media terminal will only discover that it cannot display it correctly when it acquires the unknown SEI message. In another case, when it comes to streaming media, a simple and quick method of slimming a bit stream is to remove all SEI messages as these are not required for decoding. This can lead to thinning of SEI messages that are important for representation.
    There is thus a general need within the technical field to efficiently operate supplementary processing information such as SEI data in order to solve or at least alleviate the problems of the prior art. SUMMARY
    It is a general objective to enable the provision of supplementary processing information in an efficient way.
    This and other goals are met by the achievements set out here.
    Briefly, one aspect of the embodiments relates to a method for providing supplementary processing information relating to or associated with encoded media content. The method involves providing a pseudo-identifier of the media codec employed to encode the media content. Supplemental processing information is also provided. The supplementary processing information defines post-decoding instructions for processing decoded media content achievable by decoding the encoded media content. The pseudo-identifier and supplementary processing information are co-organized into a file in order to trigger decoding abort of the media content encoded by a legacy media terminal that does not recognize the pseudo-identifier. Correspondingly, the co-organization of the pseudo-identifier and the supplementary processing information in the file enables decoding of the media content using the media codec and enables post-decoding processing of the decoded media content using the supplementary processing information by a media terminal that recognizes the pseudo-identifier.
    Another aspect relates to a device for providing supplementary processing information. The device includes an identifier provider to provide the pseudo-identifier of the media codec. An information provider is implemented in the device to provide the supplementary processing information defining the post-decoding instructions. The device also includes a file organizer to co-organize the pseudo-identifier and supplementary processing information in a file in such a way as to enable decoding abort of media content encoded by a legacy media terminal that does not recognize the pseudo. -identifier. A media terminal that recognizes the pseudo-identifier, however, will be able to decode the encoded media content using the media codec and further process the media content following decoding based on the supplementary processing information.
    A further aspect of the embodiments relates to a method for processing media content. The method involves receiving a file including the pseudo-identifier and supplementary processing information. If the retrieved pseudo-identifier of the file is not recognized, decoding of the encoded media content is aborted so that the media content will not be presented to a user. However, if the pseudo-identifier is recognized, a media codec to use to decode the encoded media content is selected based on a codec identifier of the retrieved media codec based on the pseudo-identifier. The encoded media content is then decoded using the selected media codec. Supplemental processing information retrieved from the file is employed to process the decoded media content to acquire the media content in a form suitable for representation to provide a media presentation to the user.
    A media terminal is exposed as an additional aspect of the embodiments. The media terminal includes a receiver for receiving the file with the pseudo-identifier and the supplementary processing information. A media terminal decode controller is configured to investigate the pseudo-identifier and determine whether the pseudo-identifier is recognizable by the media terminal or not. If the pseudo-identifier is not recognized, the decoding controller controls a decoder to abort or stop decoding the encoded media content. If the pseudo-identifier is instead recognized, the decoding controller selects a media codec to employ to decode the encoded media content based on a retrieved codec identifier based on the pseudo-identifier. The decoder is then configured to decode the encoded media content using the selected media codec to produce decoded media content. This decoded media content is processed by a content processor based on the supplementary processing information retrieved from the file to acquire the media content in a form suitable for representation.
    The embodiments provide an efficient way of providing supplementary processing information that is to be used in connection with post-decoding processing of media content to acquire the media content in a form that enables correct representation and presentation to a user. Legacy media terminals that do not understand and therefore cannot use the supplementary processing information are prevented from representing the media content without any post-decoding processing rather than aborting the decoding. Decoding abort is enabled by using a pseudo-identifier instead of the true codec identifier as the media codec identifier. Legacy media endpoints are therefore prevented from representing media content. This is important since the representation would otherwise lead to perceptible visual and/or auditory artifacts by a user as no post-decoding processing of the media content using the supplementary processing information was conducted by the legacy media terminals lacking the ability to use the supplemental processing information. BRIEF DESCRIPTION OF THE DRAWINGS
    The invention, together with its additional aims and advantages, can be better understood by referring to the following description taken along with the accompanying drawings, in which:
    Figure 1 is a flowchart illustrating a method for providing supplementary processing information according to an embodiment;
    Figure 2 is a comparison of media content processing according to prior art and according to an embodiment;
    Figure 3 is a flowchart illustrating an embodiment of the co-organization step in Figure 1;
    Figure 4 is a schematic overview of a media container file according to an embodiment;
    Figure 5 is a schematic overview of a sample inbox according to an embodiment;
    Figure 6 is a schematic overview of an embodiment of organizing supplementary processing information into sample inboxes of a media track;
    Figure 7 is an overview of a communication network to which embodiments can be implemented;
    Figure 8 is a schematic block diagram of a device for providing supplementary processing information according to an embodiment;
    Figure 9 is a flowchart of a method for processing media content according to an embodiment;
    Figure 10 is a flowchart of further steps of the method in Figure 9 according to an embodiment;
    Figure 11 is a flowchart of further steps of the method in Figure 9 according to another embodiment;
    Figure 12 is a schematic block diagram of a media terminal according to an embodiment; and
    Figure 13 is a flowchart illustrating another embodiment of the co-organization step in Figure 1. DETAILED DESCRIPTION
    Throughout the drawings, the same reference numbers are used for similar or corresponding elements.
    Embodiments generally relate to media content processing and in particular to the provision and use of supplementary processing information that can be employed by media terminals and players to process and/or represent decoded media content.
    The embodiments provide efficient ways to communicate such supplementary processing information to media terminals. The communication of supplemental processing information allows media terminals that recognize the supplemental processing information to retrieve and use it to process the media content before or during performance. However, the supplemental processing information is advantageously communicated in such a way that legacy media terminals that do not recognize the supplemental processing information will not represent the media content. This is preferred since such representation of media content would lead to visual and/or auditory artifacts perceived by the user as no post-decoding processing based on any supplementary processing information was conducted by the legacy media terminals.
    Supplemental Processing Information (SPI) or Supplemental Enhancement Information (SEI) as it is also denoted in the art defines post-decoding instructions that can be used by media terminals and players to process decoded media content to which the supplemental processing information is associated. These post-decoding instructions can therefore provide information and instructions to the media terminals on how they should process the data in order to provide a correct representation of the media content during a media session. Supplemental processing information could be mandatory for media terminals in terms of omitting the processing of decoded media content based on the supplementary processing information prevents the media terminal from representing the media content or the media terminal may represent the content media, but the resulting media presentation will have various more or less severe visual and/or auditory artifacts that will be perceived by the user. Supplemental processing information may also be optional in terms of not being required to represent the media content, but will enhance representation, perhaps enhancing the visual and/or auditory effects of the media presentation or seeking to facilitate operations in the media presentation.
    For example, AVC specifies SEI messages that can be used to improve decoding and VCL representation of an AVC bit stream. This information is designed in such a way that it is not mandatory for the decoding process, that is, SEI contains information that is not necessary to decode the VCL and can be discarded by conforming decoders.
    Although SEI is not crucial to the decoding process, it can specify actions required for the subsequent representation process. One example is the stereo video information SEI message which indicates that the encoded frames/fields of the video sequence consist of pairs of images forming stereo vision content. A more recent example is the "checkerboard" SEI message of spatially interleaved frames. It tells the decoder that the output decoded image contains samples of a frame consisting of multiple spatially distinct interleaved frames and how to properly deinterleave the samples for display purposes.
    If a legacy media terminal tries to decode and represent an AVC bitstream containing these SEI messages without understanding their meaning, it will discard them and decode the bitstream without knowing that the decoded images make no sense without further deinterleaving. Directly producing a 2D display, the result will be full of artifacts and certainly not what the content provider intended.
    The AVC file format defines AVC storage in file formats based on the basic ISO media file format, such as MP4 and 3GP. The AVC bitstreams contained in such files can include SEI messages, which can be useful for the decoding process. However, there is no constraint on the types of SEI messages that can be included and there is no signaling mechanism that currently informs the player that the AVC bitstream may contain SEI that is required for correct representation of the decoded data. Even if the player analyzes the AVC bitstream and looks for SEI messages, it cannot tell whether any of the detected SEI messages impose actions outside the decoding mesh unless it recognizes them and their meaning. Discarding the seemingly harmless but unknown SEI can thus lead to unexpected results. Consequently, a prior art file must be made with specific knowledge of terminal display capabilities to ensure correct reproduction. New display features added by supplementary processing information, such as SEI messages outside the decoding mesh, cannot be done in a backward compatible manner today.
    Media content as set out here relates to media data that can be communicated to a media terminal for decoding and representation therein to provide a media presentation to a user. Media content can then be video content or data that is played and presented on a display screen. Alternatively, or in addition, media content can be audio or data content that is played and can be heard by a user using a speaker.
    Figure 1 is a flowchart of a method for providing supplementary processing information relating to encoded media content according to an embodiment. The method starts at step S1, where a pseudo-identifier of a media codec is provided. The media codec defines the media encoder employed to encode relevant media content. Step SI provides a pseudo-identifier instead of the actual codec identifier of the media codec. Pseudo-identifier should be interpreted here as a pseudonym for the codec identifier. The pseudo-identifier is further selected to be recognizable and can be interpreted by some media terminals, but it is not recognized and cannot be interpreted by other media terminals, denoted legacy media terminals here.
    A next step S2 provides supplementary processing information, abbreviated to SPI in the figures. The supplemental processing information defines post-decoding instructions for processing decoded media content achievable by decoding the encoded media content associated with the supplemental processing information.
    The pseudo-identifier and supplementary processing information are co-organized at step S3 into a file in such a way as to enable decoding abort or prevent decoding of media content encoded by a legacy media terminal that does not recognize the pseudo-identifier . Thus, the pseudo-identifier preferably activates the legacy media terminal to abort decoding the media content and thereby not represent media content. However, the co-organization in the file is further conducted in such a way as to enable a media terminal that recognizes the pseudo-identifier to decode the encoded media content using the media decoder defined by the media codec and to process in post-decoding the decoded media content using the supplemental processing information.
    Thus, co-organizing the pseudo-identifier and the supplementary processing information in the file in step SI will prevent a legacy media terminal from decoding the encoded media content or enabling the media terminal to abort decoding the encoded media content when retrieves the pseudo-identifier. The reason for this is that the legacy media terminal does not recognize and therefore cannot use the supplementary processing information. This means that although the legacy media terminal can decode the encoded media content, it cannot correctly perform post-decode processing of the decoded media content using the supplemental processing information. This will lead to problems, which the embodiments solve by using the pseudo-identifier to trigger a decode abort before impersonation can be initiated on the legacy media endpoint.
    Figure 2 schematically illustrates this concept. In this illustrative example, video content is provided with restrictions in terms of requiring supplementary processing information in order to enable a correct representation and display of the video data to the user. For example, the supplementary processing information could define which video frames of the encoded video data or content correspond to a left view and which video frames correspond to a right view by stereo or 3D representation. The supplementary information could then define that every odd frame of video content represents a first view (left or right), while every even frame of video content represents a second view (right or left). According to the prior art, an old terminal, i.e. legacy media terminal, will correctly decode encoded media content. However, since it does not recognize the supplementary processing information, it will simply ignore the information. This will result in an incorrect playback or representation of video content with jumping between left and right views every frame since the legacy media terminal is not aware of the particular interleaving of video frames from different views. The media presentation will therefore be highly unattractive to a viewer user and will be full of visual artifacts due to the jump of vision to every frame.
    In the prior art, media decoding and representation is conducted on legacy media terminals, although legacy media terminals do not recognize the supplementary processing information needed in order to achieve correct media presentation.
    The pseudo-identifier and the co-organization of the pseudo-identifier and the supplementary processing information in the file according to the embodiments solve this problem with the prior art by enabling abort or stop decoding of the encoded media content for these media terminals. legacy that do not recognize the pseudo-identifier. This means that the media content is preferably never represented on the legacy media terminal and the user can instead be informed that their media terminal does not support representation of the present media content.
    This concept is illustrated schematically on the right in Figure 2. Thus, when a legacy media terminal, ie the old terminal in the figure, receives the file with the pseudo-identifier and the supplementary processing information, the legacy media terminal retrieves the pseudo-identifier. The media terminal then concludes that it does not recognize the pseudo-identifier and therefore stops decoding the media content.
    In a particular embodiment, some additional or external information could be provided along with the file to increase the media terminal's self-awareness. For example, such meta-information could, for example, be a call to inform the media terminal that its decoder is obsolete to decode the bitstream and that some update is needed. This is further illustrated in the figure, where the media terminal could look for codec updates in order to efficiently operate the file, recognize the pseudo-identifier and process decoded media content using the supplementary processing information. The media terminal can hereby be configured to request support or updates from some pre-configured external source.
    The pseudo-identifier can therefore be thought of as a means to hide the true media codec codec identifier from those legacy media terminals that will not recognize the supplementary processing information. Thus concealing the codec identifier by this means prevents the legacy mobile terminals from decoding and thereby representing the media content.
    Co-organizing the pseudo-identifier and the supplementary processing information is preferably performed in such a way that the media terminal will only retrieve the supplementary processing information from the file if it recognizes and can interpret the pseudo-identifier. This effectively prevents legacy media terminals from trying to retrieve and interpret supplementary processing information that they do not recognize.
    Embodiments will now be described further with respect to different particular implementation examples.
    Figure 3 is a flowchart illustrating the co-organization step in Figure 1 according to a particular embodiment. The method continues from step S2 of Figure 1. A next step S10 generates and provides a media container file. The media container file can be considered as a complete input package that preferably includes, in addition to the media content per se, information and instructions required by the media terminals to perform the decoding and representation of the media content. The ISO basic media file format can be advantageously employed as the file format for the media container file, including various storage formats derived from or based on the ISO media base file format, such as the AVC file format. The AVC file format instead specifies how H.264 (MPEG-4 AVC) is taken in various file formats derived from the ISO base media file format, for example the MP4 and 3GP file format.
    The media container file generated in step S10 includes at least one media track defining the media content and at least one media content container or box including the encoded media content. An example of such a media file container 1 is illustrated in Figure 4. The media container file 1 includes, in this example, a respective media track 20, 30 for each media content to be represented during a media presentation. Consequently, a first media track 20 defines video content and a second media track 30 defines corresponding audio content. Media tracks 20, 30 are preferably arranged in a film container ('moov') or box 10, which is a container for the metadata relating to the media content. The figure also illustrates the media data container ('mdat') or box 40, which is the container for the current media data for the media presentation.
    A next step Sll of the method in Figure 2 provides a sample input box in the media container file. This sample input box sets the pseudo-identifier. An example of such a sample input box might look like:

    VisualSampleEntry and AVCSampleEntry define that media content is video content. The sample input box is designed such that it has a virtual base: SampleEntry (format) extends Box (format). What is in the "format" field is undefined and is left up to your derived input to name it a value. For example, in the case of video, we have VisualSampleEntry(codingname) extend SampleEntry(codingname). In the present case, "codingname" corresponds to 'resv' which represents the pseudo-identifier. Thus, instead of defining the particular media codec employed to encode the media content, such as 'avcl', the present embodiment replaces the media codec's codec identifier with the pseudo-identifier 'resv'.
    A further processing information box is provided in the sample inbox at step S12. The supplemental processing information box includes the supplemental processing information applicable to the media content. This supplementary processing information box can be arranged in the media container file according to various embodiments. In a first embodiment, the supplementary processing information box is provided directly in the sample inbox. The sample input box can then be extended to read as, where SeilnformationBox corresponds to the supplementary processing information box:

    In an alternative embodiment, the supplementary processing information box is provided deeper into the constrained schema information box rather than directly into the sample inbox. The constrained schema information box could then be defined as:

    In such a case, the supplementary processing information box may be contained in the layout information box. The inclusion of the supplementary processing information box in the schema information box could, in a particular embodiment, be conditional on using 'aSEP, ie, AVC SEI, as the schema type.
    In any case, the supplementary process information box may, in a particular embodiment, be defined as:

    Thus, in this particular embodiment, the supplementary processing information can be divided into mandatory supplementary processing information (requiredSEI_ID), which must be used by the media terminal during post-decoding processing of the media content and optional supplementary processing information ( Not required SEIID). The latter is not mandatory, but will generally improve or enhance the media presentation or representation of media content if employed by the media terminal. numRequiredSEIs and numNotRequiredSEIs define the number of versions of mandatory and optional supplemental processing information present in the media container file and relative to the particular media content. requiredSEI_ID takes the "payloadType" value of a supplementary processing information message, such as a SEI message, present in the media stream and which is deemed necessary by the author or file creator for correct representation or reproduction. NotrequiredSEI ID correspondingly assumes the "payloadType" value of a supplementary processing information message (SEI message) which is not deemed necessary by the file author for correct representation.
    The supplementary processing information box therefore preferably only includes respective identifiers of the supplementary processing information messages that can be used with respect to the particular media content. The particular supplementary processing information messages referred to by these identifiers can then be provided elsewhere in the media container file, which is further described here.
    The next step S13 in Figure 3 provides a media format box in the sample input box. The media format box includes the media codec codec identifier.
    In a particular embodiment, the original sample input type, ie codec identifier, is contained in an OriginalFormatBox, representing the media format box. The OriginalFormatBox can then be contained within the RestrictedSchemelnfoBox mentioned above:

    So, in the example above, "datajformat" represents the actual media codec identifier and can take various values such as 'avd', 'mp4v', and so on depending on the particular media codec.
    Figure 5 schematically illustrates the various containers or boxes that can be used to define the pseudo-identifier, codec identifier, and supplementary processing information. In this particular embodiment, sample input box 80 has its traditional four-character code, such as 'avcl', replaced by the pseudo-identifier represented by 'resv' in the figure denoting "restricted video". In this example, the sample input box 80 in turn includes, among others, the constrained scheme information box 70, denoted 'rinf in the figure. Restricted scheme box information 70 includes original format box 60 (denoted 'frma' in the figure) with codec identifier 62, such as 'avcl'. In a particular embodiment, the constrained scheme information box 70 also includes the supplementary processing information box 50 denoted 'seii' in the figure. Supplemental processing information box 50 includes or lists identifiers 52 of required or mandatory supplementary processing information messages and identifiers 54 of required or optional supplementary processing information messages.
    Figure 6 is a figure representing a media track 20 of the media container file. This figure illustrates that media track 20 can be divided into multiple sample input boxes 80, where each such sample input box 80 is associated with a respective portion of the encoded media content contained in the file's media content box. of media container. In such a case, supplementary processing information 50 may be present in sample input boxes 80 as mentioned previously. In a particular embodiment, supplementary processing information 50 defining instructions applicable to a subsequent post-decoding portion of the encoded media content is provided in a sample inbox associated with a previous portion of the encoded media content. This previous portion of the encoded content then has a decoding time preceding the decoding time of the subsequent portion of the encoded media content. Thus, the supplementary processing information 50 defining post-decoding instructions relating to the media content portion k, k+1, k+2 is provided in the sample input box 80 relating to the media content portion of k-1, k, k+1. In such a case, the media terminal will have time to identify and provide the supplementary processing information and its post-decoding instructions defined when the relevant media content version was decoded by the media terminal. Thus, any delay in decoding and post-decoding processing will therefore be reduced.
    However, in most embodiments, sample inboxes 80 and media tracks are typically sent together before the actual data present in the media data container is transmitted. Thus, for these embodiments, there is generally no need to provide supplementary processing information 50 defining post-decoding instructions relating to media content k in sample inbox 80 relating to the media content portion k-1. In clear contrast, the supplementary processing information 50 could instead be provided in the sample inbox relating to the media content portion k.
    The actual post-decoding instructions defined by the supplementary processing information can be provided in the current bit stream. Thus, post-decoding instructions, such as SEI messages, can be encapsulated in NAL units and are then placed deep in the bit stream, where they are contained by the media data box. Another place that the actual post-decoding instructions can be located is in some special box, for example ViewScalabilitylnfoSEIBox, which just makes a copy of the current post-decoding instructions in the bitstream.
    Additional embodiments of the media container file could replace traditional sample input, such as AVC sample input, in such a way that legacy media terminals will not be able to decode it. The new sample input is then defined in a similar way to the original syntax, but with the addition of the supplementary processing information box. In such a case, this new sample entry will match the pseudo-identifier of the media codec. For example, the sample inbox and pseudo-identifier could be denoted 'avcr' for AVC with constraints:

    In yet another embodiment, a new version of AVCDecoderConfigurationRecord is defined and consequently a new value for configurationversion in AVCDEcoderConfigurationRecord is used. This is done so that legacy media terminals that do not use supplemental processing information will not be able to decode the file.
    In an optional case, the new definition of AVCDecoderConfigurationRecord also contains information about required and/or optional supplementary processing information. An example of this syntax is shown below:


    In an alternative embodiment, the file in which the pseudo-identifier and supplementary process information are co-organized does not need a media container file. Figure 13 illustrates this embodiment. The method continues from step S2 of Figure 1. A next step S50 involves generating a Session Description Protocol (SDP) file and co-organizing the pseudo-identifier and supplementary processing information as different attribute lines in the SDP file. . As is known in the art, Real Transport Protocol (RTP) is designed for real-time data transmission. Different payload formats are defined for different media codecs and these govern the packaging rules. Side information is also typically sent in the SDP file between the sender and receiver so that both parties are aware of the state of the media content being transmitted establishing an SDP offer/response model.
    The SDP file can then be modified to have a first attribute line including the pseudo-identifier and a second attribute line including the supplementary processing information. Legacy media terminals that cannot use the supplemental processing information for post-decoding processing of the media content will not recognize the pseudo-identifier retrieved from the SDP file. Decoding of media content is hereby stopped and prevented.
    In a particular embodiment, the pseudo-identifier is used as a new token type instead of the old token, such as replacing H264 with H264-RES, which means H.264 with restrictions. Also new stream property parameters are defined declaring which restrictions are applied. These flow property parameters correspond to supplementary processing information that can be specified as required/not required SEI messages. An example is shown below:

    When a receiver of legacy media terminal receives the above SDP offer, pseudo-identifier H264-RES stops erroneous decoding since the legacy media terminal will reject this offer in advance. Only media terminals that recognize H264-RES will accept the offer and therefore can correctly decode and represent media content with the supplementary processing information. These media terminals will thus map or translate the pseudo-identifier H264-RES to the codec identifier, ie, H264. This means that these media terminals have been configured to understand the pseudo-identifier and conduct the required identifier mapping or translation.
    The embodiments set out in the foregoing were mainly exemplified with H.264/AVC. However, this should only be seen as illustrative but not limiting examples. For example, equivalent embodiments with H.264/AVC extensions, such as Expandable Video Coding (SVC) and Multiview Video Coding (MVC), among others, work in a very similar way by extending their specific configuration registers and/ or sample input types in a similar manner.
    The embodiments therefore enable detection of the presence of supplementary processing information by a media terminal without examining the bit stream. In clear contrast, it is sufficient to investigate the relevant part of the media container file or the SDP file or offer as exemplified above. The supplementary processing information will furthermore advantageously only be seen by those media terminals that can interpret and use the supplementary processing information. A legacy media terminal will therefore not attempt to display a decoded bitstream that would have required additional post-processing before rendering.
    Figure 7 is a schematic overview of a portion of a communication network in which embodiments can be implemented. The communication network is exemplified by a radio-based communication network providing, among others, communication services to mobile terminals 200, 210 and other user devices, through base stations 310 or network nodes. The communication network also includes a media server 300 implemented within or connected to such a base station 310 for providing media content to one or more requesting media terminals, such as mobile terminals 200, 210. The figure illustrates a possible embodiment implementing a device for providing supplementary processing information 100, denoted SPI provision device in the figures. This device 100 is configured to generate a file 1 as set forth herein including the pseudo-identifier and the supplementary processing information. The file 1 is then transmitted to the mobile terminal 200, 210 via the connected base station 310. In the figure, mobile terminal 210 could represent a legacy media terminal, while mobile terminal 200 indicates a media terminal that recognizes the pseudo-identifier and can use supplementary processing information.
    Device 100 does not necessarily have to be deployed on media server 300, but can alternatively be deployed on another network node of the communication network. A distributed implementation on multiple such network nodes is also possible and within the scope of the embodiments.
    Figure 8 is a schematic block diagram of the device for providing supplementary processing information 100 according to an embodiment. Device 100 includes an identifier provider 110 configured to provide the pseudo-identifier of the media codec employed to encode the media content. An information provider 130 is configured to provide supplementary processing information representing post-decoding instructions for processing decoded media content achievable by decoding the encoded media content. Device 100 also includes a file organizer 120 configured to co-organize the pseudo-identifier from the identifier provider 110 and the supplementary processing information from the information provider 130 into a file. Co-organizing in the file is, as discussed here, conducted by the file organizer 120 in such a way as to enable decoding abort of the content encoded by a legacy media terminal that does not recognize the pseudo-identifier. However, the co-organization enables decoding of the encoded media content using the media codec and enables post-decoding processing of the decoded media content using the supplementary processing information by a media terminal that recognizes the pseudo-identifier.
    Information provider 130 could provide both required supplementary processing information representing mandatory post-decoding instructions to process the decoded media content and not required supplementary processing information representing optional post-decoding instructions to process the decoded media content.
    In a particular embodiment, file organizer 120 is configured to generate a media container file including at least one media track defining the media content and at least one media content box including the encoded media content. The identifier provider 110 could then provide a sample inbox in the media container file. The sample input box defines the pseudo-identifier and includes a media format box including the codec identifier as described previously. Information provider 130 preferably provides a supplementary processing information box including the supplementary processing information in the sample inbox. In a particular embodiment, the information provider 130 provides the supplementary processing information by defining post-decoding instructions applicable to a portion of media content k in the sample inbox associated with a portion of prior media content k-1 having a decoding time preceding the decoding time of the media content portion k. In another embodiment, the information provider 130 provides the supplementary processing information by defining post-decoding instructions applicable to the media content portion k in the sample inbox associated with the media content portion k.
    In an alternative embodiment, the file organizer 120 is configured to generate an SDP file including the pseudo-identifier from the identifier provider 110 as a first attribute line and the supplementary processing information from the information provider 130 as a second attribute line. attribute.
    Units 110 to 130 of device 100 may be implemented or provided as hardware or a combination of hardware and software. In the case of a software-based implementation, a computing program product implementing device 100 or a portion thereof includes software or a computing program running on a general purpose or specially adapted computer, processor or microprocessor. The software includes computer program code elements or software code portions illustrated in Figure 8. The program may be stored in whole or in part, or on one or more satisfactory non-transient computer readable media or such data storage media. such as magnetic disks, CD-ROMs, DVD disks, USB memories, hard disks, magneto-optical memory, in RAM or volatile memory, in ROM or flash memory, as firmware, or on a data server.
    Device 100 can be implemented advantageously within or with respect to a media server 300 as illustrated in Figure 7.
    Figure 9 is a flowchart illustrating a method for processing media content at a media terminal according to an embodiment. The method starts at step S20, where a file including the pseudo-identifier and the supplementary processing information is received. A next step S21 checks whether the pseudo-identifier, abbreviated to PID in the figures, is recognizable. If the pseudo-identifier is not recognizable, the method continues at step S22, where media content decoding is aborted or prevented. As discussed in the foregoing, the encoded media content can either be provided in the same file as the pseudo-identifier and the supplementary processing information or provided separately in step S20 or in another step (not exposed). If the pseudo-identifier is recognizable instead, the method continues from step S21 to step S23. This step S23 selects a media codec to decode the media content based on a retrieved codec identifier based on the pseudo-identifier. The next step S24 decodes the media content using the selected media codec. The decoded media content is processed in step S25 based on the supplementary processing information in order to acquire media content in a form suitable for representation, such as display or reproduction.
    The method could then terminate. Alternatively, media content could be divided into different media content portions where supplementary processing information could be associated and applicable to such different media portions. In such a case, steps S24 and S25 are repeated for each such portion of media content.
    Figure 10 is a flowchart illustrating additional steps of the method in Figure 9 according to an embodiment. The method continues from step S20 of Figure 9, which involves receiving a media container file including at least one media track defining the media content, at least one media content box including the encoded media content, and at least one sample inbox. The sample input box defines the pseudo-identifier and includes a supplemental processing information box with the supplemental processing information and a media format box with the codec identifier.
    A next step S30 retrieves the pseudo-identifier from the sample inbox and investigates in step S21 of Figure 9 whether the pseudo-identifier is recognizable. If this is true, the method continues at step S31, where the codec identifier is retrieved from the media format box. The method then continues to step S23 of Figure 9, where the media codec is selected based on the retrieved codec identifier.
    Figure 11 is a flowchart illustrating additional steps of the method in Figure 9 according to another embodiment. The method continues from step S20 in Figure 9, which involves receiving an SDP file including a first attribute line including the pseudo-identifier and a second attribute line including the supplementary processing information. A next step S40 reads the first attribute line in the SDP file to identify the pseudo-identifier. The method then continues at step S21 in Figure 9 to investigate whether the pseudo-identifier is recognizable. If this is true, the method continues at step S41. Step S41 maps or translates the pseudo-identifier into the codec identifier. The method then continues to step S23 of Figure 9, where the media codec is selected based on the mapped codec identifier.
    If the supplementary processing information defines post-decoding instructions that allow the identification of which video frames belong to a left view and which video frames belong to a stereo or 3D representation right view, the processing step in Figure 9 could including identifying the video frames of the encoded video data corresponding to the left view and the video frames corresponding to the right view based on the supplementary processing information.
    Figure 12 is a schematic block diagram of a media terminal 200 according to an embodiment. Media terminal 200 is exemplified by a mobile terminal in the figure. However, this should only be seen as an illustrative example. Media terminal could be any entity or device or aggregation of multiple devices that has decoding and representation capability. A single such device could be a mobile terminal such as a mobile phone or laptop, a computer, a TV-to-TV converter or any other media processing device. The decoding and rendering functionality may be present in different devices which are then capable of conducting wired or wireless communication with each other. Media Terminal therefore also encompasses such a distributed implementation embodiment.
    Media terminal 200 includes a receiver 210 exemplified by a transceiver (TX/RX) in the figure. Receiver 210 is configured to receive a file including the pseudo-identifier and supplementary processing information. A decode controller 220 is configured to retrieve the pseudo-identifier from the file and verify that the pseudo-identifier is recognizable. If not recognized by the decoder controller 220, the decoder controller 220 controls the decoder 230 to abort the decoding of the encoded media content or stops the decoder 230 from starting the decoding process. If the decoder controller 220 instead recognizes the pseudo-identifier, it selects the media codec 235 to be used by the decoder 230 to decode the media content. This codec selection is conducted based on a codec identifier determined by the decoding controller 220 using the pseudo-identifier.
    The decoder 230 then decodes the encoded media content using the selected media codec 235 to obtain decoded MB media content. The decoded media content is further processed by a content processor 240 based on the supplementary processing information retrieved from the file to acquire media content in a form suitable for representation. The media terminal 200 could then have a media player (not illustrated), which represents the media content rendered for display on a display screen 250. Alternatively or in addition, the media player could play audio content at a high -speaker (not illustrated) of the 200 media terminal.
    In a particular embodiment, receiver 210 receives a media container file with at least one media track defining the media content, at least one media content box including the media content and preferably the defined post-decoding instructions by the supplementary processing information. At least one sample input box is also included in the media container file and defines the pseudo-identifier and includes the supplemental processing information box with the supplemental processing information and the media format box with the codec identifier . In such a case, the media terminal 200 preferably includes an identifier provider 225 configured to retrieve the pseudo-identifier from the sample inbox. The identifier provider 225 preferably also retrieves, if the pseudo-identifier is recognizable by the decoding controller 220, the codec identifier of the media format box.
    In an alternative embodiment, the receiver 210 receives an SDP file including a first attribute line including the pseudo-identifier and a second attribute line including the supplementary processing information. The identifier provider 225 is then configured to read the first attribute row to retrieve the pseudo identifier and then, if the pseudo identifier is recognizable by the decode controller 220, map the pseudo identifier to the codec identifier of the media codec 235 using preconfigured mapping data stored in the media terminal 200. Such preconfigured mapping data could be a list including the pseudo-identifiers that are recognized by the decode controller 220 and the matching codec identifiers.
    Units 210 to 240 of the media terminal 200 can be implemented or provided as hardware or a combination of hardware and software. In the case of a software-based implementation, a computer program product implementing the media terminal 200 or a portion thereof includes software or a computer program running on a general purpose or specially adapted computer, processor or microprocessor. The software includes computer program code elements or software code portions illustrated in Figure 12. The program may be stored in whole or in part, on one or more unsatisfactory computer readable media or such data storage media. such as magnetic disks, CD-ROMs, DVD disks, USB memories, hard disks, magneto-optical memory, in RAM or volatile memory, in ROM or flash memory, such as firmware, or on a data server.
    The embodiments described above are to be understood as some illustrative examples of the present invention. It will be understood by those skilled in the art that various modifications, combinations and changes can be made to the embodiments without departing from the scope of the present invention. In particular, different part solutions in different embodiments can be combined in other configurations, where technically possible. The scope of the present invention, however, is defined by the appended claims.
    权利要求:
    Claims (8)
    [0001]
    1. A method for providing supplementary processing information (52, 54) relating to encoded media content, said method comprising: providing (S2) supplementary processing information (52, 54) defining post-decoding instructions for media terminals ( 200, 210) to process decoded media content obtained by decoding said encoded media content to provide a correct rendering of said decoded media content during a media session, characterized by: providing (S1) a pseudo-identifier 'resv' as a pseudonym of a media codec employed to encode said media content; co-organize (S3) said pseudo-identifier 'resv' and said supplementary processing information (52, 54) in a file (1) to enable decoding abort of said media content encoded by a legacy media terminal (210) which does not recognize said 'resv' pseudo-identifier and to enable decoding of said encoded media content using said media codec and post-decoding processing of said decoded media content using said supplementary processing information (52, 54) by a media terminal (200) recognizing said pseudo-identifier 'resv', wherein co-organizing (S3) said pseudo-identifier 'resv' and said supplementary processing information comprises: generating (S10) a media container file ( 1) comprising at least one media track (20) defining said media content and at least one media data box (40) comprising said encoded media content; providing (S11), in said media container file (1), a sample input box (80) with said pseudo-identifier 'resv' as sample input type and comprising a restricted schema information box (70 ); providing (S12), in said restricted scheme information box (70) a supplementary processing information box (50) comprising said supplementary processing information (52, 54); and providing (S13), in said restricted scheme information box (70), an original format box (60) comprising a codec identifier (62) of said media codec as representative of an original sample input type.
    [0002]
    2. Method according to claim 1, characterized in that said media track (20) comprises multiple sample inboxes (80), each being associated with a respective portion of said encoded media content contained in said at least one media data box (40), wherein providing (S12) said supplementary processing information box (50) comprises providing (S12) a supplementary processing information box (50) defining post-decoding instructions applicable to a subsequent portion of said encoded media content in a sample inbox (50) associated with a previous portion of said encoded media content having a decoding time preceding a decoding time of said subsequent portion of said encoded content. encoded media.
    [0003]
    3. Method according to claim 1 or 2, characterized in that providing (S2) said supplementary processing information (52, 54) comprises providing (S2) at least one supplementary processing information (52) representing instructions post-decoding instructions for processing said decoded media content and supplementary processing information (54) representing optional post-decoding instructions for processing said decoded media content.
    [0004]
    4. Device (100) for providing supplementary processing information (52, 54) relating to encoded media content, said device (100), said device (100) comprising: an information provider (130) configured to provide information of supplementary processing (52, 54) defining post-decoding instructions for media terminals (200, 210) to process decoded media content obtained by decoding said encoded media content to provide a correct rendering of said decoded media content during a media session, characterized by: an identifier provider (110) configured to provide a pseudo-identifier 'resv', as a pseudonym of a media codec employed to encode said media content; a file organizer (120) configured to co-organize said pseudo-identifier 'resv' provided by said identifier provider (110) and said supplementary processing information (52, 54) provided by said information provider (130) into one file (1) to enable abort decoding of said media content encoded by a legacy media terminal (210) which does not recognize said pseudo-identifier 'resv' and to enable decoding of said media content encoded using said media codec and post-decoding processing of said decoded media content using said supplementary processing information (52, 54) by a media terminal (200) recognizing said pseudo-identifier 'resv', wherein. said file organizer (120) is configured to generate a media container file (1) comprising at least one media track (20) defining said media content and at least one media data box (40) comprising said content of encoded media; said identifier provider (110) is configured to provide in said media container file (1), a sample input box (80), with said pseudo-identifier 'resv' as sample input type and comprising a sample input box. restricted scheme information (70) and providing, in said restricted scheme information box (70) an original format box (60) comprising a codec identifier (62) of said media codec as representing an input type of original sample; and said information provider (130) is configured to provide, in said restricted schema information box (70), a supplementary processing information box (50) comprising said supplementary processing information (52, 54).
    [0005]
    5. Device according to claim 4, characterized in that said media track (20) comprises multiple sample inboxes (80), each being associated with a respective portion of said encoded media content contained in said at least one media data box (40), wherein said information provider (130) is configured to provide a supplementary processing information box (50) defining post-decoding instructions applicable to a subsequent portion of said content. encoded media in a sample inbox (50) associated with a previous portion of said encoded media content having a decoding time preceding a decoding time of said subsequent portion of said encoded media content.
    [0006]
    6. Device according to claim 4 or 5, characterized in that said information provider (130) is configured to provide at least one of supplementary processing information (52) representing mandatory post-decoding instructions to process said decoded media content and supplementary processing information (54) representing optional post-decoding instructions for processing said decoded media content.
    [0007]
    7. Media server (300), characterized in that it comprises a device (100) for providing supplementary processing information (52, 54) as defined in any one of claims 4 to 6.
    [0008]
    8. Network node (310), characterized in that it comprises a media server (300) as defined in claim 7.
    类似技术:
    公开号 | 公开日 | 专利标题
    BR112012009148B1|2021-05-25|method and device for providing supplementary processing information relating to encoded media content
    WO2015096540A1|2015-07-02|Bitstream generation and processing methods and devices and system
    JP2011528868A|2011-11-24|Method and apparatus for track and track subset grouping
    KR20210080607A|2021-06-30|Method and apparatus for encoding media data comprising generated content
    US10827170B2|2020-11-03|Method and device for coding POC, method and device for decoding POC, and electronic equipment
    KR102000868B1|2019-07-16|Apparatus, method, and computer readable medium for encoding
    CN105960804A|2016-09-21|Signal transmission and reception apparatus and signal transmission and reception method for providing trick play service
    WO2019121963A1|2019-06-27|Prioritized transmission of predetermined portions of encapsulated media content
    TW201909647A|2019-03-01|Enhanced area orientation encapsulation and visual independent high-efficiency video writing media data file
    JP2022017182A|2022-01-25|Methods, appliances, and computer programs for encapsulating media data into media files.
    GB2587364A|2021-03-31|Method, device, and computer program for encapsulating media data into a media file
    US20210105492A1|2021-04-08|Method and apparatus for storage and signaling of sub-sample entry descriptions
    WO2020187564A1|2020-09-24|Method and apparatus for encapsulating derived tracks
    CN113170239A|2021-07-23|Method, apparatus and computer program for encapsulating media data into a media file
    WO2020182524A1|2020-09-17|Method, device, and computer program for signaling available portions of encapsulated media content
    同族专利:
    公开号 | 公开日
    RU2012120698A|2013-12-10|
    IN2012DN03081A|2015-07-31|
    CN102648629B|2015-05-06|
    US20170201762A1|2017-07-13|
    AU2010308600A1|2012-05-17|
    MX2012004534A|2012-06-08|
    US10397595B2|2019-08-27|
    RU2540804C2|2015-02-10|
    JP2013509084A|2013-03-07|
    HUE037388T2|2018-08-28|
    AU2010308600B2|2014-09-25|
    CN104661115B|2020-08-04|
    ES2671520T3|2018-06-06|
    EP2491720A1|2012-08-29|
    PL2491720T3|2018-08-31|
    ZA201202647B|2013-06-26|
    EP2491720B1|2018-03-07|
    US10085036B2|2018-09-25|
    WO2011049517A1|2011-04-28|
    CN104661115A|2015-05-27|
    EP3346711A1|2018-07-11|
    DK2491720T3|2018-06-06|
    US9609368B2|2017-03-28|
    CA2778064A1|2011-04-28|
    EP2491720A4|2013-12-11|
    US20120201521A1|2012-08-09|
    BR112012009148A2|2017-08-08|
    NZ599303A|2014-06-27|
    NO2491720T3|2018-08-04|
    CA2778064C|2019-06-04|
    US20180352244A1|2018-12-06|
    CN102648629A|2012-08-22|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    JP4183587B2|2003-09-12|2008-11-19|シャープ株式会社|Image recording device|
    US7650036B2|2003-10-16|2010-01-19|Sharp Laboratories Of America, Inc.|System and method for three-dimensional video coding|
    JP2005167584A|2003-12-02|2005-06-23|Matsushita Electric Ind Co Ltd|Apparatus for protecting media stream in mp4 file format|
    US7689051B2|2004-04-15|2010-03-30|Microsoft Corporation|Predictive lossless coding of images and video|
    CN100433159C|2004-09-23|2008-11-12|北京凯诚高清电子技术有限公司|Multisystem type information recording medium, information recording and playing method and device|
    US8046479B2|2006-11-07|2011-10-25|Telefonaktiebolaget Lm Ericsson |Media channel management|
    US8463019B2|2007-07-19|2013-06-11|JVC Kenwood Corporation|Pseudo 3D image generation device, image encoding device, image encoding method, image transmission method, image decoding device, and image decoding method|
    CN101802823A|2007-08-20|2010-08-11|诺基亚公司|Segmented metadata and indexes for streamed multimedia data|
    US8400497B2|2007-09-07|2013-03-19|Samsung Electronics Co., Ltd|Method and apparatus for generating stereoscopic file|
    US8171541B2|2007-12-18|2012-05-01|Nokia Corporation|Enabling provider network inter-working with mobile access|
    US20090199100A1|2008-02-05|2009-08-06|Samsung Electronics Co., Ltd.|Apparatus and method for generating and displaying media files|
    US9088817B2|2009-06-16|2015-07-21|Lg Electronics Inc.|Broadcast transmitter, broadcast receiver and 3D video processing method thereof|
    CN104717479B|2009-10-13|2017-05-03|Lg电子株式会社|Broadcast receiver and 3D video data processing method thereof|CN102291580B|2011-07-28|2013-03-20|南京联慧通信技术有限公司|Video transmission method based on Android system|
    EP2758885A4|2011-08-29|2015-10-28|Cinsay Inc|Containerized software for virally copying from one endpoint to another|
    US9584793B2|2012-04-09|2017-02-28|Intel Corporation|Signaling three-dimensional video information in communication networks|
    US10789631B2|2012-06-21|2020-09-29|Aibuy, Inc.|Apparatus and method for peer-assisted e-commerce shopping|
    US9607330B2|2012-06-21|2017-03-28|Cinsay, Inc.|Peer-assisted shopping|
    US20140079116A1|2012-09-20|2014-03-20|Qualcomm Incorporated|Indication of interlaced video data for video coding|
    US9161039B2|2012-09-24|2015-10-13|Qualcomm Incorporated|Bitstream properties in video coding|
    GB2542282B|2013-10-22|2018-08-01|Canon Kk|Method, device, and computer program for encapsulating partitioned timed media data in a server|
    US10028002B2|2013-12-16|2018-07-17|Samsung Electronics Co., Ltd.|Server device for sharing contents, client device, and method for sharing contents|
    US9716900B2|2014-06-20|2017-07-25|Qualcomm Incorporated|Extensible design of nesting supplemental enhancement informationmessages|
    US10979691B2|2016-05-20|2021-04-13|Qualcomm Incorporated|Circular fisheye video in virtual reality|
    US10536721B2|2017-01-09|2020-01-14|Qualcomm Incorporated|Restricted scheme design for video|
    US10999605B2|2017-01-10|2021-05-04|Qualcomm Incorporated|Signaling of important video information in file formats|
    US10805650B2|2017-03-27|2020-10-13|Qualcomm Incorporated|Signaling important video information in network video streaming using mime type parameters|
    US10735806B2|2018-09-07|2020-08-04|Disney Enterprises, Inc.|Configuration for detecting hardware-based or software-based decoding of video content|
    法律状态:
    2018-03-27| B15K| Others concerning applications: alteration of classification|Ipc: H04N 7/00 (2011.01) |
    2019-01-08| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|
    2020-03-03| B06U| Preliminary requirement: requests with searches performed by other patent offices: procedure suspended [chapter 6.21 patent gazette]|
    2021-03-30| B15K| Others concerning applications: alteration of classification|Free format text: A CLASSIFICACAO ANTERIOR ERA: H04N 7/00 Ipc: H04N 19/13 (2014.01), H04N 19/176 (2014.01), H04N |
    2021-04-06| B09A| Decision: intention to grant [chapter 9.1 patent gazette]|
    2021-05-25| B16A| Patent or certificate of addition of invention granted|Free format text: PRAZO DE VALIDADE: 20 (VINTE) ANOS CONTADOS A PARTIR DE 18/10/2010, OBSERVADAS AS CONDICOES LEGAIS. PATENTE CONCEDIDA CONFORME ADI 5.529/DF |
    优先权:
    申请号 | 申请日 | 专利标题
    US25312809P| true| 2009-10-20|2009-10-20|
    US61/253,128|2009-10-20|
    PCT/SE2010/051119|WO2011049517A1|2009-10-20|2010-10-18|Provision of supplemental processing information|
    [返回顶部]