Title of Invention	METHOD FOR DISPLAYING MULTIMEDIA DATA
Abstract	An apparatus and a method for displaying audio and video data, and a storage medium for storing the method thereon. The apparatus for displaying audio and video data constituting multimedia data described in MPV format, ascertains whether an asset selected by a user comprises a single video data and at least one or more audio data, extracts reference information to display the video data and the audio data and then displays the extracted video data, using the reference information, and extracts at least one or more audio data from the reference information and then sequentially displays them according to a predetermined method while the video data is being displayed.

Title of Invention

METHOD FOR DISPLAYING MULTIMEDIA DATA

Abstract

An apparatus and a method for displaying audio and video data, and a storage medium for storing the method thereon. The apparatus for displaying audio and video data constituting multimedia data described in MPV format, ascertains whether an asset selected by a user comprises a single video data and at least one or more audio data, extracts reference information to display the video data and the audio data and then displays the extracted video data, using the reference information, and extracts at least one or more audio data from the reference information and then sequentially displays them according to a predetermined method while the video data is being displayed.

Full Text	FORM 2 THE PATENTS ACT, 1970 (39 of 1970) & THE PATENTS RULES, 2003 COMPLETE SPECIFICATION (See section 10, rule 13) 'APPARATUS AND METHOD FOR DISPLAYING AUDIO AND VIDEO DATA, AND STORAGE MEDIUM RECORDING THEREON A PROGRAM TO EXECUTE THE DISPLAYING METHOD' SAMSUNG ELECTRONICS CO. LTD., 416 Maetan-dong, Yeongtong-gu, Suwon-si, Gyeonggi-do, Republic of Korea, The following specification particularly describes the invention and the manner in which it is to be performed. Technical Field [1] The present invention relates to an apparatus and a method for displaying audio and video data (hereinafter referred to as 'AV data') and a storage medium on which a program to execute the displaying method is recorded, and more particularly to, management of audio and video data among multimedia data in the format of Multi-Photo Video or MusicPhoto Video (both of which are hereinafter referred to as 'MPV') and provision of the same to users. Background Art [2] MPV is an industrial standard specification dedicated to multimedia titles, published by the Optical Storage Technology Association (hereinafter referred to as 'OSTA'), an international trade association established by optical storage makers in 2002. Namely, MPV is a standard specification to provide a variety of music, photo and video data more conveniently or to manage and process the multimedia data. The definition of MPV and other standard specifications are available for use through the official web site ("www.osta.org') of OSTA. [3] Recently, media data comprising digital pictures, video, digital audio, text and the like are processed and played by means of personal computers (PC). Devices for playing the media content, e.g., digital cameras, digital camcorders, digital audio players (namely, digital audio data playing devices such as Moving Picture Experts Group Layer-3 Audio (MP3), Window Media Audio (WMA) and so on) have been in frequent use, and various kinds of media data have been produced in large quantities accordingly. [4] However, personal computers have mainly been used to manage multimedia data produced in large quantities; in this regard file-based user experience has been requested. In addition, when multimedia data is produced on a specified product, attributes of the data, data playing sequences, and data playing methods are produced depending upon the multimedia data. If they are accessed by the personal computers, the attributes are lost and only the source data is transferred. In other words, there is a -2.- WO 2005/029490 PCT7KR2004/002309 very weak interoperability relative to data and attributes of the data between household electric goods, personal computers and digital content playing devices. 5. An example of the weak interoperability will be described. A picture is captured using a digital camera, and data such as the sequence for an attribute slide show, determined by use of a slideshow function to identify the captured picture on the digital camera, time intervals between pictures, relations between pictures whose attributes determined using a panorama function are taken, and attributes determined using a consecutive photoing function are stored along with actual picture data as the source data. At this time, if the digital camera transfers pictures to a television set using an AV cable, a user can see multimedia data whose respective attributes are represented. However, if the digital camera is accessed via a personal computer using a universal serial bus (USB), only the source data is transferred to the computer and the pictures' respective attributes are lost. 6. As described above, it is shown that the interoperability of the personal computer for metadata such as attributes of data stored in the digital cameral is very weak. Or, there is no interoperability of the personal computer to the digital camera. 7. To strengthen the interoperability relative to data between digital devices, the standardization for MPV has been in progress. 8. MPV specification defines Manifest, Metadata and Practice to process and play sets of multimedia data such as digital pictures, video, audio, etc. stored in storage median (or device) comprising an optical disk, a memory card, a computer hard disk, or exchanged according to the Internet Protocol (IP). 9. The standardization for MPV is currently overseen by the OSTA (Optical Storage Technology Association) and I3A (International Imaging Industry Association), and the MPV takes an open specification and mainly desires to make it easy to process, exchange and play sets of digital pictures, video, digital audio and text and so on. 10. MPV is roughly classified into MPV Core-Spec (0.90WD) and Profile. 11. The core is composed of three basic factors such as Collection, Metadata and Identification. 12. The Collection has Manifest as a Root member, and it comprises Metadata, Album, MarkedAsset and AssetList, etc. The Asset refers to multimedia data described according to the MPV format, being classified into two kinds: Simple media asset (e.g., digital pictures, digital audio, text, etc.) and Composite media asset (e.g., digital picture combined with digital audio (StillWithAudio), digital pictures photoed consecutively (StillMultishotSequence), panorama digital pictures 3 WO 2005/029490 PCT/KR2004/002309 (StillPanoramaSequence), etc.). FIG. 1 illustrates examples of StillWithAudio, Still-MultishotSequence, and StillPanoramaSequence. [13] Metadata adopts the format of extensible markup language (XML) and has five kinds of identifiers for identification. [14] 1. LastURL is a path name and file name of a concerned asset (Path to the object), [15] 2. Instance lD is an ID unique to each asset (unique per object: e.g., Exif 2.2), [16] 3. Document ID is identical to both source data and modified data, [17] 4. Content ID is created whenever a concerned asset is used for a specified purpose, and [18] 5. id is a local variable within metadata. [19] There are seven profiles: Basic profile, Presentation profile, Capture/Edit profile, Archive profile, Internet profile, Printing profile and Container profile. [20] MPV supports management of various file associations by use of XML metadata so as to allow various multimedia data recorded on storage media to be played. Especially, MPV supports JPEG (Joint Photographic Experts Group), MP3, WMA(Windows Media Audio), WMV (Windows Media Video), MPEG-1 (Moving Picture Experts Group-1), MPEG-2, MPEG-4, and digital camera formats such as AVI (Audio Video Interleaved) and Quick Time MJPEG (Motion Joint Photographic Experts Group) video. MPV specification-adopted discs are compatible with BO9660 level 1, Joliet, and also multi-session CD (Compact Disc), DVD (Digital Versatile Disc), memory cards, hard discs and Internet, thereby allowing users to manage and process more various multimedia data. Disclosure of Invention Technical Problem [21] However, new formats of various multimedia data not defined in MPV format specification, namely new formats of assets are in need, and an addition of a function to provide the multimedia data is on demand. Technical Solution [22] Accordingly, the present invention is proposed to provide formats of new multimedia data in addition to various formats of multimedia data defined in the current MPV formats, and increase the utilization of various multimedia data by proposing a method to provide multimedia data described according to MPV formats to users in a variety of ways. [23] According to an exemplary embodiment of the present invention, there is provided an apparatus for displaying audio and video data constituting multimedia data 4 WO 2005/029490 PCT/KR2004/002309 described in MPV format, wherein the apparatus ascertains whether an asset selected by a user comprises a single audio data and at least one or more video data, extracts reference information to display the audio data and the video data and then displays the audio data extracted, by use of the reference information, and extracts at least one or more video data from the reference information and then sequentially displays them according to a predetermined method while the audio data is being output. The displaying operation may allow the video data to be displayed according to information on display time to determine the playback times of respective video data while the audio data is being displayed and information on volume control to adjust the volume generated when the audio data and the video data are being played. [24] According to another exemplary embodiment of the present invention, there is provided an apparatus for displaying audio and video data constituting multimedia data described in MPV format, wherein the apparatus ascertains whether an asset selected by a user comprises a single video data and at least one or more audio data, extracts reference information to display the video data and the audio data and then displays the video data extracted, using the reference information, and extracts at least one or more audio data from the reference information and then sequentially displays them according to a predetermined method while the video data is being displayed. The displaying method may allow the audio data to be displayed according to information on display time to determine the playback times of respective audio data while the video data is being displayed and information on volume control to adjust the volume generated when the audio data are being played. [25] According to a another exemplary embodiment of the present invention, there is provided a method for displaying audio and video data constituting multimedia data described in MPV format, comprising ascertaining whether an asset selected by a user comprises a single audio data and at least one or more video data, extracting reference information to display the audio data and the video data, extracting and displaying the audio data using the reference information, and extracting and sequentially displaying at least one or more video data from the reference information according to a predetermined method while the audio data is being displayed. [26] The displaying method may allow the video data to be displayed according to in- formation on display time to determine the playback times of respective video data while the audio data is being displayed and information on volume control to adjust the volume generated when the audio data and the video data are being played. At this time, the display time information may comprise information on start time when the -5- WO 2005/029490 PCT/KR2004/002309 video data starts to be played" and information on playback time to indicate the playback time of the video data. [27] The extraction and sequential display step comprises synchronizing first time in- formation to designate the time for playing the audio data and second time information to designate the time for playing the at least one or more video data, extracting first volume control information to adjust the volume generated while the audio data is being played and second volume control information to adjust the volume while the at least one or more video data are being displayed, and supplying the audio data and the video data through a display medium by use of the time information and the volume control information. [28] According to a still further exemplary embodiment of the present invention, there is provided a method for displaying audio and video data constituting multimedia data described in MPV format, comprising ascertaining whether an asset selected by a user comprises single video data and at least one or more audio data, extracting reference information to display the video data and the audio data, extracting and displaying the video data using the reference information, and extracting and sequentially displaying at least one or more audio data from the reference information according to a predetermined method while the video data is being displayed. [29] The displaying method may allow the audio data to be output according to in- formation on display time to determine the playback times of respective audio data while the video data is being displayed and information on volume control to adjust the volume generated when the video data and the audio data are being played. At this time, the display time information may comprise information on start time when the audio data starts to be played" and information on playback time to indicate the playback time of the video data. [30] The extraction and sequential display step may comprise synchronizing first time information to designate the time for playing video data and second time information to designate the time for playing the at least one or more audio data, extracting first volume control information to adjust the volume generated while the video data is being played and second volume control information to adjust the volume while the at least one or more audio data are being displayed, and supplying the video data and the audio data through a display medium by use of the time information and the volume control information. [31] According to a still further exemplary embodiment of the present invention, there is provided a storage medium recording thereon a program for displaying multimedia 6 WO 2005/029490 PCT7KR2004/002309 data described in MPV format, wherein the program ascertains whether an asset selected by a user comprises a single audio data and at least one or more video data, extracts reference information to display the audio data and the video data and then displays the audio data extracted, using the reference information, and extracts at least one or more video data from the reference information and then displays them sequentially according to a predetermined method while the audio data is being output. [32] According to a still further exemplary embodiment of the present invention, there is provided a storage medium recording thereon a program for displaying multimedia data described in MPV format, wherein the program ascertains whether an asset selected by a user comprises a single video data and at least one or more audio data, extracts reference information to display the video data and the audio data and then displays the video data extracted, using the reference information, and extracts at least one or more audio data from the reference information and then sequentially displays them according to a predetermined method while the video data is being displayed. Description of Drawings [33] FIG. 1 is an exemplary view illustrating different kinds of assets described in a MPV specification; [34] FIG. 2 is an exemplary view schematically illustrating a structure of an 'Au- dio With Video' asset according to an aspect of the present invention; [35] FIG. 3 is an exemplary view illustrating a element according to an aspect of the present invention; [36] FIG. 4 is an exemplary view illustrating an element according to an aspect of the present invention; [37] FIG. 5 is an exemplary view illustrating a element according to an aspect of the present invention; [38] FIG. 6 is an exemplary view illustrating a element according to an aspect of the present invention; [39] FIG. 7 is an exemplary view illustrating a element according to an aspect of the present invention; [40] FIG. 8 is an exemplary view illustrating an element according to an aspect of the present invention; [41] FIG. 9 is an exemplary diagram illustrating a type of an element according to an aspect of the present invention; [42] FIG. 10 is an exemplary diagram illustrating a structure of an 'VideoWithAudio' asset according to an aspect of the present invention; -7~ WO 2005/029490 PCT/KR2004/002309 [43] FIG. 11 is an exemplary view illustrating an element according to an aspect of the present invention; [44] FIG. 12 is an exemplary view illustrating an element according to an aspect of the present invention; [45] FIG. 13 is an exemplary view illustrating element according to an aspect of the present invention; [46] FIG. 14 is an exemplary diagram illustrating a type of an element according to an aspect of the present invention; [47] FIG. 15 is an exemplary view illustrating an AudioRefGroup according to an aspect of the present invention; [48] FIG. 16 is an exemplary view illustrating a VideoRefGroup according to an aspect of the present invention; [49] FIG. 17 is a flow chart illustrating a process of playing the 'Audio With Video' asset according to an aspect of the present invention; and [50] FIG. 18 is a block diagram of an apparatus for displaying audio and video data, according to an exemplary embodiment of the present invention. Mode for Invention [51] Hereinafter, an apparatus and a method for displaying audio and video data, which are based on MPV formats, according to an aspect of the present invention, will be described in more detail with reference to the accompanying drawings. [52] In the present invention, XML is used to provide multimedia data according to MPV format. Thus, the present invention will be described according to XML schema. [53] More various multimedia data are provided herein by proposing new assets of 'Au- dioWithVideo' and 'VideoWithAudio' not provided by OSTA. To describe the new assets, the following terms are used: 'smpv' and 'mpv' refer to a 'namespace' in XML, wherein the former indicates a namespace relative to a new element proposed in the present invention and the latter indicates a namespace relative to an element proposed by the OSTA. The definitions and examples of these new assets will be described. [54] 1. Audio with Video asset [55] This 'AudioWithVideo' asset comprises a combination of a single audio asset with at least one or more video assets. To represent this asset in XML, it may be referred to as an element of . Where a user enjoys at least one or more moving picture contents while listening to a song, this will constitute an example of this asset. At this time, the time interval to play multiple moving picture contents can be controlled, and also the volume from the moving picture contents and that from the 8 WO 2005/029490 PC17KR2004/002309 song can be controlled. [56] The audio asset and the video asset are treated as elements in XML documents, that is, XML files. The audio asset may be represented as and and the video asset may be represented as and . [57] The element indicates a part of the audio asset. As a sub-element of the , , , can be defined. Among the three sub-elements, a value of at least one sub-element must be designated. [58] sub-element may be defined as type='xs:long' minOccurs='0'/>, indicating the start time relative to a part of the entire play time of the audio asset, referenced in the unit of seconds. Given no value thereto, the start time is calculated as in [SMPV:start] = [SMPV:stop] - [SMPV:dur] based on and . Where values of or are not designated, the value of is 0. [59] sub-element may be defined as type='xs:long' minOccurs='0'/>, indicating the stop time relative to a part of the entire play time of the audio asset referenced in the unit of seconds. Given no value thereto, the stop time is calculated as in [SMPV:stop] = [SMPV:start] + [SMPV:dur] based on and . Where a value of is not designated but a value of is designated, the value of is equal to the stop time of an asset referenced. Where a value of is not designated but is designated, the value of is equal to the value of . [60] SMPV:dur> sub-element may be defined as type='xs:long' minOccurs='07>, indicating the actual play time of the audio asset referenced. Where a value of is not given, tins time is calculated as in [SMPV:dur] = [SMPV:stop] - [SMPV:start]. [61 ] The element indicates a part of the video asset. The same method of defining the element can be employed in defining the element. [62] FIG. 2 is an exemplary view schematically illustrating a structure of 'Au- dio With Video' asset according to an aspect of the present invention. [63] Referring to this figure, the elements respectively having 'mpv' or 'smpv' as namespace. [64] Elements having 'mpv' as namespace are described in the official homepage of 9 WO 2005/029490 PCT/KR2004/002309 OSTA (www.osta.org) proposing MPV specification, description thereof will be omitted herein. Accordingly, only elements having 'smpv' as namespace will be described below. [65] (1) [66] This element references the element. [67] (2) [68] This element references the element. [69] (3) [70] This element references the element, which is illustrated in FIG. 3. [71] (4) [72] This element references the FIG. 4. [73] (5) [74] A value of this element indicates the play time of respective video data, being represented in the unit of seconds and indicating a relative time value. The play time may be presented in decimal points. Where a value of this element is not set, it is regarded that the play time is not set, and thus, the total play time of any concerned video data is assumed to be equal to the value of the element. [75] The total play time of any concerned video data may be determined depending upon a reference type of the video data referenced in the video asset. [76] Namely, the total play time of a concerned video data is equal to the total play time of the video data referenced when the reference type is 'VideoRef.' Where the reference type is 'VideoPartRef,' it is possible to obtain the total play time of the concerned video data using an attribute value of the element referenced. Where the reference type is 'AudioPartRef,' the reference type relative to the audio data should be identified in the referenced element. To be specific, where the reference type relative to the audio data is 'AudioRef,' the total play time of the concerned video data is equal to the total play time of the audio data, and where the reference type relative to the audio data is 'AudioPartRef,' the total play time of the concerned video data can be obtained by an attribute value of the referenced element Further, where the reference type is 'VideoWithAudioRef,' only the video asset is extracted from the element, and the total play time of the video data referenced as 'VideoRef in the extracted video asset is regarded as the total play time of the concerned video data. 10 WO 2005/029490 PCT/KR2004/002309 [77] A value of the element will be described in brief. [78] VideoDurSeq = (";") (1) clock-value = ( \| ) (2) unknown-dur=the empty string (3) seconds = (.) (4) [79] Formula (1) means that a value of the element is represented as 'clock-value,' and play times of respective video type are identified by means of';' where there are two or more video data. [80] Formula (2) means that 'clock-value' in Formula (1) is indicated as 'seconds' or unknown-dur.' [81] Formula (3) means that Unknown-dur' in Formula (2) indicates no setting of 'clock- value.' [82] Formula (4) means that 'seconds' in Formula (2) is indicated as a decimal and playback time of the concerned video data can be indicated by means of a decimal point. [83] For example, where 'clock-value' is '7.2,' this means that the playback time of the concerned video data is 7.2 seconds. As another example, where 'clock-value' is '2:10.9,' this means that there are two video data concerned, one of which is played for 2 seconds and the other of which is placed for 10.9 seconds. As a further example, where 'clock-value' is ';5.6,' this means that there are two video data concerned, one of which is played for the total playback time of the concerned content because its playback time is not set, and the other of which is played for 5.6 seconds. FIG. 5 illustrates the element. [84] (6) [85] A value of element indicates a point in time when each of video data starts to play back. The point in time is in the unit of seconds, indicating a relative time value based on the start times of the respective video data. The playback start time may be indicated as a decimal point. For example, where a value of the element is not set, the value is assumed to be 0 seconds. Namely, the concerned video data is played from the playback start time thereof. If the value of element is larger than the total playback time of the concerned video data, it causes the concerned video data to play after the playback thereof ends: in this case, the value of -11 WO 2005/029490 PCT/KR2004/002309 element is assumed to be 0. [86] If element and element are both defined within dioWithVideo> element, the value of summing element and element should be equal to or less than the total playback time of the concerned video data. If not so, the value of element becomes the deduction of the value of element from the total playback time of the concerned video data. FIG. 6 illustrates the element. [87] (7) [88] A value of element indicates the volume size of the concerned video data by percentage. Thus, where the value of element is 0, the volume of the concerned video data becomes 0. If the value of element is not set, die concerned video data is played with the volume as originally set. [89] While a plurality of video data are played, values of the element, as many as the played video data, are set. However, if a single value is set, all of the video data played are played with the volume having the single value as set. FIG. 7 illustrates die element. [90] (8) [91] A value of indicates the volume size of the concerned audio data in percentage. When the value of element is not set, it is assumed to be 100. FIG. 8 illustrates the element [92] FIG. 9 is an exemplary diagram illustrating a type of an element according to an aspect of the present invention. [93] An exemplary method for providing an asset of above-described elements will be described. [94] [Example 1] A0007 V 1205 V1206 ;3 ^SMPV:AudioWithVideo> 12 WO 2005/029490 PCT/KR2004/002309 [95] Example 1 illustrates a method of playing the audio asset referenced as 'A0007' and two video assets referenced as 'V1205' and 'V1206' respectively. In this example, since a value of element is not set with respect to the video asset whose value is referenced as 'V1205,' the value is assumed to be 0 seconds. Namely, the video asset referenced as 'V1205' is being played from the point in time when the audio asset referenced as 'A0007 starts to play to the time when the video asset referenced as 'V1206' starts to play. Meanwhile, since a value of the element is set to be 3 with respect to the video asset whose value is referenced as 'VI206,' the video asset referenced as 'VI206' is being played in three seconds after the point in time when the video asset referenced as 'V1206' starts to play. [96] [Example 2] A0001 V 1001 V 1002 V 1003 2;; 10 ;3;0 50 50 [97] Example 2 illustrates a method of playing an AudioWithVideo asset using one audio asset referenced as 'A0001' and three video assets referenced as 'V1001,' 'V1002' and 'V1003' respectively. In this example, the video asset referenced as 'V0001' is played for two seconds. The video asset referenced as 'V1002' starts to play after playback of the video asset referenced as 'V1001' ends and after three seconds have passed since the video asset referenced as 'V1001' starts to play. The video asset referenced as 'V1003' is being played for ten seconds after playback of the video asset -13- WO 2005/029490 PCT/KR2004/002309 referenced as 'V1002' ends. [98] The three video assets are played with the volume sizes of 50% of their original volumes, and the audio asset is also played with the volume size of 50% of its original volume. [99] [Example 3] A0001 VPl 001 AV 1002 [100] 2. 'VideoWithAudio' Asset [101] This 'VideoWithAudio' asset comprises a combination of a single video asset with at least one or more audio assets. To represent this asset in XML, it may be referred to as an element of . The audio asset and the video asset are treated as elements in XML documents. The audio asset may be represented as or

Full Text

FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
'APPARATUS AND METHOD FOR DISPLAYING AUDIO AND VIDEO
DATA, AND STORAGE MEDIUM RECORDING THEREON A PROGRAM TO
EXECUTE THE DISPLAYING METHOD'
SAMSUNG ELECTRONICS CO. LTD., 416 Maetan-dong, Yeongtong-gu, Suwon-si, Gyeonggi-do, Republic of Korea,
The following specification particularly describes the invention and the manner in which it is to be performed.

Technical Field
[1] The present invention relates to an apparatus and a method for displaying audio
and video data (hereinafter referred to as 'AV data') and a storage medium on which a program to execute the displaying method is recorded, and more particularly to, management of audio and video data among multimedia data in the format of Multi-Photo Video or MusicPhoto Video (both of which are hereinafter referred to as 'MPV') and provision of the same to users.
Background Art
[2] MPV is an industrial standard specification dedicated to multimedia titles,
published by the Optical Storage Technology Association (hereinafter referred to as 'OSTA'), an international trade association established by optical storage makers in 2002. Namely, MPV is a standard specification to provide a variety of music, photo and video data more conveniently or to manage and process the multimedia data. The definition of MPV and other standard specifications are available for use through the official web site ("www.osta.org') of OSTA.
[3] Recently, media data comprising digital pictures, video, digital audio, text and the
like are processed and played by means of personal computers (PC). Devices for playing the media content, e.g., digital cameras, digital camcorders, digital audio players (namely, digital audio data playing devices such as Moving Picture Experts Group Layer-3 Audio (MP3), Window Media Audio (WMA) and so on) have been in frequent use, and various kinds of media data have been produced in large quantities accordingly.
[4] However, personal computers have mainly been used to manage multimedia data
produced in large quantities; in this regard file-based user experience has been requested. In addition, when multimedia data is produced on a specified product, attributes of the data, data playing sequences, and data playing methods are produced depending upon the multimedia data. If they are accessed by the personal computers, the attributes are lost and only the source data is transferred. In other words, there is a
-2.-

WO 2005/029490 PCT7KR2004/002309
very weak interoperability relative to data and attributes of the data between household electric goods, personal computers and digital content playing devices.
5. An example of the weak interoperability will be described. A picture is captured using a digital camera, and data such as the sequence for an attribute slide show, determined by use of a slideshow function to identify the captured picture on the digital camera, time intervals between pictures, relations between pictures whose attributes determined using a panorama function are taken, and attributes determined using a consecutive photoing function are stored along with actual picture data as the source data. At this time, if the digital camera transfers pictures to a television set using an AV cable, a user can see multimedia data whose respective attributes are represented. However, if the digital camera is accessed via a personal computer using a universal serial bus (USB), only the source data is transferred to the computer and the pictures' respective attributes are lost.
6. As described above, it is shown that the interoperability of the personal computer for metadata such as attributes of data stored in the digital cameral is very weak. Or, there is no interoperability of the personal computer to the digital camera.
7. To strengthen the interoperability relative to data between digital devices, the standardization for MPV has been in progress.
8. MPV specification defines Manifest, Metadata and Practice to process and play sets of multimedia data such as digital pictures, video, audio, etc. stored in storage median (or device) comprising an optical disk, a memory card, a computer hard disk, or exchanged according to the Internet Protocol (IP).
9. The standardization for MPV is currently overseen by the OSTA (Optical Storage Technology Association) and I3A (International Imaging Industry Association), and the MPV takes an open specification and mainly desires to make it easy to process, exchange and play sets of digital pictures, video, digital audio and text and so on.
10. MPV is roughly classified into MPV Core-Spec (0.90WD) and Profile.
11. The core is composed of three basic factors such as Collection, Metadata and Identification.
12. The Collection has Manifest as a Root member, and it comprises Metadata, Album, MarkedAsset and AssetList, etc. The Asset refers to multimedia data described
according to the MPV format, being classified into two kinds: Simple media asset
(e.g., digital pictures, digital audio, text, etc.) and Composite media asset (e.g., digital picture combined with digital audio (StillWithAudio), digital pictures photoed consecutively (StillMultishotSequence), panorama digital pictures
3

WO 2005/029490 PCT/KR2004/002309

(StillPanoramaSequence), etc.). FIG. 1 illustrates examples of StillWithAudio, Still-MultishotSequence, and StillPanoramaSequence.
[13] Metadata adopts the format of extensible markup language (XML) and has five
kinds of identifiers for identification.
[14] 1. LastURL is a path name and file name of a concerned asset (Path to the object),
[15] 2. Instance lD is an ID unique to each asset (unique per object: e.g., Exif 2.2),
[16] 3. Document ID is identical to both source data and modified data,
[17] 4. Content ID is created whenever a concerned asset is used for a specified purpose,
and
[18] 5. id is a local variable within metadata.
[19] There are seven profiles: Basic profile, Presentation profile, Capture/Edit profile,
Archive profile, Internet profile, Printing profile and Container profile.
[20] MPV supports management of various file associations by use of XML metadata
so as to allow various multimedia data recorded on storage media to be played. Especially, MPV supports JPEG (Joint Photographic Experts Group), MP3, WMA(Windows Media Audio), WMV (Windows Media Video), MPEG-1 (Moving Picture Experts Group-1), MPEG-2, MPEG-4, and digital camera formats such as AVI (Audio Video Interleaved) and Quick Time MJPEG (Motion Joint Photographic Experts Group) video. MPV specification-adopted discs are compatible with BO9660 level 1, Joliet, and also multi-session CD (Compact Disc), DVD (Digital Versatile Disc), memory cards, hard discs and Internet, thereby allowing users to manage and process more various multimedia data.
Disclosure of Invention
Technical Problem
[21] However, new formats of various multimedia data not defined in MPV format
specification, namely new formats of assets are in need, and an addition of a function to provide the multimedia data is on demand.
Technical Solution
[22] Accordingly, the present invention is proposed to provide formats of new
multimedia data in addition to various formats of multimedia data defined in the current MPV formats, and increase the utilization of various multimedia data by proposing a method to provide multimedia data described according to MPV formats to users in a variety of ways.
[23] According to an exemplary embodiment of the present invention, there is provided
an apparatus for displaying audio and video data constituting multimedia data

4

WO 2005/029490

PCT/KR2004/002309

described in MPV format, wherein the apparatus ascertains whether an asset selected by a user comprises a single audio data and at least one or more video data, extracts reference information to display the audio data and the video data and then displays the audio data extracted, by use of the reference information, and extracts at least one or more video data from the reference information and then sequentially displays them according to a predetermined method while the audio data is being output. The displaying operation may allow the video data to be displayed according to information on display time to determine the playback times of respective video data while the audio data is being displayed and information on volume control to adjust the volume generated when the audio data and the video data are being played.
[24] According to another exemplary embodiment of the present invention, there is
provided an apparatus for displaying audio and video data constituting multimedia data described in MPV format, wherein the apparatus ascertains whether an asset selected by a user comprises a single video data and at least one or more audio data, extracts reference information to display the video data and the audio data and then displays the video data extracted, using the reference information, and extracts at least one or more audio data from the reference information and then sequentially displays them according to a predetermined method while the video data is being displayed. The displaying method may allow the audio data to be displayed according to information on display time to determine the playback times of respective audio data while the video data is being displayed and information on volume control to adjust the volume generated when the audio data are being played.
[25] According to a another exemplary embodiment of the present invention, there is
provided a method for displaying audio and video data constituting multimedia data described in MPV format, comprising ascertaining whether an asset selected by a user comprises a single audio data and at least one or more video data, extracting reference information to display the audio data and the video data, extracting and displaying the audio data using the reference information, and extracting and sequentially displaying at least one or more video data from the reference information according to a predetermined method while the audio data is being displayed.
[26] The displaying method may allow the video data to be displayed according to in-
formation on display time to determine the playback times of respective video data while the audio data is being displayed and information on volume control to adjust the volume generated when the audio data and the video data are being played. At this time, the display time information may comprise information on start time when the
-5-

WO 2005/029490 PCT/KR2004/002309
video data starts to be played" and information on playback time to indicate the playback time of the video data.
[27] The extraction and sequential display step comprises synchronizing first time in-
formation to designate the time for playing the audio data and second time information to designate the time for playing the at least one or more video data, extracting first volume control information to adjust the volume generated while the audio data is being played and second volume control information to adjust the volume while the at least one or more video data are being displayed, and supplying the audio data and the video data through a display medium by use of the time information and the volume control information.
[28] According to a still further exemplary embodiment of the present invention, there
is provided a method for displaying audio and video data constituting multimedia data described in MPV format, comprising ascertaining whether an asset selected by a user comprises single video data and at least one or more audio data, extracting reference information to display the video data and the audio data, extracting and displaying the video data using the reference information, and extracting and sequentially displaying at least one or more audio data from the reference information according to a predetermined method while the video data is being displayed.
[29] The displaying method may allow the audio data to be output according to in-
formation on display time to determine the playback times of respective audio data while the video data is being displayed and information on volume control to adjust the volume generated when the video data and the audio data are being played. At this time, the display time information may comprise information on start time when the audio data starts to be played" and information on playback time to indicate the playback time of the video data.
[30] The extraction and sequential display step may comprise synchronizing first time
information to designate the time for playing video data and second time information to designate the time for playing the at least one or more audio data, extracting first volume control information to adjust the volume generated while the video data is being played and second volume control information to adjust the volume while the at least one or more audio data are being displayed, and supplying the video data and the audio data through a display medium by use of the time information and the volume control information.
[31] According to a still further exemplary embodiment of the present invention, there
is provided a storage medium recording thereon a program for displaying multimedia
6

WO 2005/029490 PCT7KR2004/002309
data described in MPV format, wherein the program ascertains whether an asset selected by a user comprises a single audio data and at least one or more video data, extracts reference information to display the audio data and the video data and then displays the audio data extracted, using the reference information, and extracts at least one or more video data from the reference information and then displays them sequentially according to a predetermined method while the audio data is being output.
[32] According to a still further exemplary embodiment of the present invention, there
is provided a storage medium recording thereon a program for displaying multimedia data described in MPV format, wherein the program ascertains whether an asset selected by a user comprises a single video data and at least one or more audio data, extracts reference information to display the video data and the audio data and then displays the video data extracted, using the reference information, and extracts at least one or more audio data from the reference information and then sequentially displays them according to a predetermined method while the video data is being displayed.
Description of Drawings
[33] FIG. 1 is an exemplary view illustrating different kinds of assets described in a
MPV specification;
[34] FIG. 2 is an exemplary view schematically illustrating a structure of an 'Au-
dio With Video' asset according to an aspect of the present invention;
[35] FIG. 3 is an exemplary view illustrating a element
according to an aspect of the present invention;
[36] FIG. 4 is an exemplary view illustrating an element
according to an aspect of the present invention;
[37] FIG. 5 is an exemplary view illustrating a element according to an
aspect of the present invention;
[38] FIG. 6 is an exemplary view illustrating a element according to an
aspect of the present invention;
[39] FIG. 7 is an exemplary view illustrating a element according to
an aspect of the present invention;
[40] FIG. 8 is an exemplary view illustrating an element according to
an aspect of the present invention;
[41] FIG. 9 is an exemplary diagram illustrating a type of an
element according to an aspect of the present invention;
[42] FIG. 10 is an exemplary diagram illustrating a structure of an 'VideoWithAudio'
asset according to an aspect of the present invention;
-7~

WO 2005/029490 PCT/KR2004/002309
[43] FIG. 11 is an exemplary view illustrating an element according to
an aspect of the present invention;
[44] FIG. 12 is an exemplary view illustrating an element
according to an aspect of the present invention;
[45] FIG. 13 is an exemplary view illustrating element according to an
aspect of the present invention;
[46] FIG. 14 is an exemplary diagram illustrating a type of an
element according to an aspect of the present invention;
[47] FIG. 15 is an exemplary view illustrating an AudioRefGroup according to an
aspect of the present invention;
[48] FIG. 16 is an exemplary view illustrating a VideoRefGroup according to an aspect
of the present invention;
[49] FIG. 17 is a flow chart illustrating a process of playing the 'Audio With Video' asset
according to an aspect of the present invention; and
[50] FIG. 18 is a block diagram of an apparatus for displaying audio and video data,
according to an exemplary embodiment of the present invention.
Mode for Invention
[51] Hereinafter, an apparatus and a method for displaying audio and video data, which
are based on MPV formats, according to an aspect of the present invention, will be described in more detail with reference to the accompanying drawings.
[52] In the present invention, XML is used to provide multimedia data according to
MPV format. Thus, the present invention will be described according to XML schema.
[53] More various multimedia data are provided herein by proposing new assets of 'Au-
dioWithVideo' and 'VideoWithAudio' not provided by OSTA. To describe the new assets, the following terms are used: 'smpv' and 'mpv' refer to a 'namespace' in XML, wherein the former indicates a namespace relative to a new element proposed in the present invention and the latter indicates a namespace relative to an element proposed by the OSTA. The definitions and examples of these new assets will be described.
[54] 1. Audio with Video asset
[55] This 'AudioWithVideo' asset comprises a combination of a single audio asset with
at least one or more video assets. To represent this asset in XML, it may be referred to as an element of . Where a user enjoys at least one or more moving picture contents while listening to a song, this will constitute an example of this asset. At this time, the time interval to play multiple moving picture contents can be controlled, and also the volume from the moving picture contents and that from the
8
WO 2005/029490 PC17KR2004/002309
song can be controlled.
[56] The audio asset and the video asset are treated as elements in XML documents,
that is, XML files. The audio asset may be represented as and and the video asset may be represented as and .
[57] The element indicates a part of the audio asset. As a sub-element of
the , , , can be defined. Among the three sub-elements, a value of at least one sub-element must be designated.
[58] sub-element may be defined as type='xs:long' minOccurs='0'/>, indicating the start time relative to a part of the entire play time of the audio asset, referenced in the unit of seconds. Given no value thereto, the start time is calculated as in [SMPV:start] = [SMPV:stop] - [SMPV:dur] based on and . Where values of or are not designated, the value of is 0.
[59] sub-element may be defined as type='xs:long' minOccurs='0'/>, indicating the stop time relative to a part of the entire play time of the audio asset referenced in the unit of seconds. Given no value thereto, the stop time is calculated as in [SMPV:stop] = [SMPV:start] + [SMPV:dur] based on and . Where a value of is not designated but a value of is designated, the value of is equal to the stop time of an asset referenced. Where a value of is not designated but is designated, the value of is equal to the value of .
[60] SMPV:dur> sub-element may be defined as type='xs:long' minOccurs='07>, indicating the actual play time of the audio asset referenced. Where a value of is not given, tins time is calculated as in [SMPV:dur] = [SMPV:stop] - [SMPV:start].
[61 ] The element indicates a part of the video asset. The same method of
defining the element can be employed in defining the element.
[62] FIG. 2 is an exemplary view schematically illustrating a structure of 'Au-
dio With Video' asset according to an aspect of the present invention.
[63] Referring to this figure, the elements respectively having 'mpv' or 'smpv' as namespace.
[64] Elements having 'mpv' as namespace are described in the official homepage of
9

WO 2005/029490 PCT/KR2004/002309

OSTA (www.osta.org) proposing MPV specification, description thereof will be omitted herein. Accordingly, only elements having 'smpv' as namespace will be described below.
[65] (1)
[66] This element references the element.
[67] (2)
[68] This element references the element.
[69] (3)
[70] This element references the element, which is illustrated in
FIG. 3.
[71] (4)
[72] This element references the FIG. 4.
[73] (5)
[74] A value of this element indicates the play time of respective video data, being
represented in the unit of seconds and indicating a relative time value. The play time may be presented in decimal points. Where a value of this element is not set, it is regarded that the play time is not set, and thus, the total play time of any concerned video data is assumed to be equal to the value of the element.
[75] The total play time of any concerned video data may be determined depending
upon a reference type of the video data referenced in the video asset.
[76] Namely, the total play time of a concerned video data is equal to the total play time
of the video data referenced when the reference type is 'VideoRef.' Where the reference type is 'VideoPartRef,' it is possible to obtain the total play time of the concerned video data using an attribute value of the element referenced. Where the reference type is 'AudioPartRef,' the reference type relative to the audio data should be identified in the referenced element. To be specific, where the reference type relative to the audio data is 'AudioRef,' the total play time of the concerned video data is equal to the total play time of the audio data, and where the reference type relative to the audio data is 'AudioPartRef,' the total play time of the concerned video data can be obtained by an attribute value of the referenced element Further, where the reference type is 'VideoWithAudioRef,' only the video asset is extracted from the element, and the total play time of the video data referenced as 'VideoRef in the extracted video asset is regarded as the total play time of the concerned video data.
10

WO 2005/029490 PCT/KR2004/002309
[77] A value of the element will be described in brief.
[78]
VideoDurSeq = (";") (1)
clock-value = ( | ) (2)
unknown-dur=the empty string (3)
seconds = (.) (4)
[79] Formula (1) means that a value of the element is represented as
'clock-value,' and play times of respective video type are identified by means of';' where there are two or more video data.
[80] Formula (2) means that 'clock-value' in Formula (1) is indicated as 'seconds' or
unknown-dur.'
[81] Formula (3) means that Unknown-dur' in Formula (2) indicates no setting of 'clock-
value.'
[82] Formula (4) means that 'seconds' in Formula (2) is indicated as a decimal and
playback time of the concerned video data can be indicated by means of a decimal point.
[83] For example, where 'clock-value' is '7.2,' this means that the playback time of the
concerned video data is 7.2 seconds. As another example, where 'clock-value' is '2:10.9,' this means that there are two video data concerned, one of which is played for 2 seconds and the other of which is placed for 10.9 seconds. As a further example, where 'clock-value' is ';5.6,' this means that there are two video data concerned, one of which is played for the total playback time of the concerned content because its playback time is not set, and the other of which is played for 5.6 seconds. FIG. 5 illustrates the element.
[84] (6)
[85] A value of element indicates a point in time when each of video data
starts to play back. The point in time is in the unit of seconds, indicating a relative time value based on the start times of the respective video data. The playback start time may be indicated as a decimal point. For example, where a value of the element is not set, the value is assumed to be 0 seconds. Namely, the concerned video data is played from the playback start time thereof. If the value of element is larger than the total playback time of the concerned video data, it causes the concerned video data to play after the playback thereof ends: in this case, the value of
-11
WO 2005/029490 PCT/KR2004/002309
element is assumed to be 0.
[86] If element and element are both defined within dioWithVideo> element, the value of summing element and element should be equal to or less than the total playback time of the concerned video data. If not so, the value of element becomes the deduction of the value of element from the total playback time of the concerned video data. FIG. 6 illustrates the element.
[87] (7)
[88] A value of element indicates the volume size of the concerned
video data by percentage. Thus, where the value of element is 0, the volume of the concerned video data becomes 0. If the value of element is not set, die concerned video data is played with the volume as originally set.
[89] While a plurality of video data are played, values of the
element, as many as the played video data, are set. However, if a single value is set, all of the video data played are played with the volume having the single value as set. FIG. 7 illustrates die element.
[90] (8)
[91] A value of indicates the volume size of the concerned audio data
in percentage. When the value of element is not set, it is assumed to be 100. FIG. 8 illustrates the element
[92] FIG. 9 is an exemplary diagram illustrating a type of an

element according to an aspect of the present invention.
[93] An exemplary method for providing an asset of above-described elements will be described.
[94]
[Example 1]
A0007
V 1205
V1206
;3 ^SMPV:AudioWithVideo>
12

WO 2005/029490 PCT/KR2004/002309
[95] Example 1 illustrates a method of playing the audio asset referenced as 'A0007' and two video assets referenced as 'V1205' and 'V1206' respectively. In this example, since a value of element is not set with respect to the video asset whose value is referenced as 'V1205,' the value is assumed to be 0 seconds. Namely, the video asset referenced as 'V1205' is being played from the point in time when the audio asset referenced as 'A0007 starts to play to the time when the video asset referenced as 'V1206' starts to play. Meanwhile, since a value of the element is set to be 3 with respect to the video asset whose value is referenced as 'VI206,' the video asset referenced as 'VI206' is being played in three seconds after the point in time when the video asset referenced as 'V1206' starts to play.
[96]
[Example 2]

A0001
V 1001
V 1002
V 1003
2;; 10
;3;0
50
50

[97] Example 2 illustrates a method of playing an AudioWithVideo asset using one
audio asset referenced as 'A0001' and three video assets referenced as 'V1001,' 'V1002' and 'V1003' respectively. In this example, the video asset referenced as 'V0001' is played for two seconds. The video asset referenced as 'V1002' starts to play after playback of the video asset referenced as 'V1001' ends and after three seconds have passed since the video asset referenced as 'V1001' starts to play. The video asset referenced as 'V1003' is being played for ten seconds after playback of the video asset
-13-

WO 2005/029490

PCT/KR2004/002309

referenced as 'V1002' ends.
[98] The three video assets are played with the volume sizes of 50% of their original
volumes, and the audio asset is also played with the volume size of 50% of its original
volume. [99]
[Example 3]

A0001
VPl 001 AV 1002

[100] 2. 'VideoWithAudio' Asset
[101] This 'VideoWithAudio' asset comprises a combination of a single video asset with
at least one or more audio assets. To represent this asset in XML, it may be referred to as an element of . The audio asset and the video asset are treated as elements in XML documents. The audio asset may be represented as or

METHOD FOR DISPLAYING MULTIMEDIA DATA

Documents:

Inventors:

PCT Conventions: