Title of Invention	APPARATUS AND METHOD FOR CREATING NEW MEDIA CONTENT IN REAL TIME
Abstract	The current invention relates to application within a mobile device for mixing or replacing live audio signal with pre-recorded audio-video signal. the mobile device is provided with an audio processing unit. the audio processing unit receives live audio signal via microphone. pre-recorded or pre-stored audio-video signal is received from media file reader via audio decoder. the audio processing unit performs editing function such as mixing, replacing or inserting live audio with pre-recorded audio signal. it also has a provision for inserting audio signal is encoded with an audio encoder and stored in a media file writer along with its corresponding video signal. thus, "talk with me" application software provides a real-time editing feature within the mobile device.

Title of Invention

APPARATUS AND METHOD FOR CREATING NEW MEDIA CONTENT IN REAL TIME

Abstract

The current invention relates to application within a mobile device for mixing or replacing live audio signal with pre-recorded audio-video signal. the mobile device is provided with an audio processing unit. the audio processing unit receives live audio signal via microphone. pre-recorded or pre-stored audio-video signal is received from media file reader via audio decoder. the audio processing unit performs editing function such as mixing, replacing or inserting live audio with pre-recorded audio signal. it also has a provision for inserting audio signal is encoded with an audio encoder and stored in a media file writer along with its corresponding video signal. thus, "talk with me" application software provides a real-time editing feature within the mobile device.

Full Text	FIELD OF INVENTION The present invention, in general, relates to the field of mobile communication. The invention describes Software Desktop application, feature phone (mobile phone) and smart phone. This invention relates to a new use case for desktop, mobile phone with audio record facility. More particularly, this invention relates to a system and method for creating new media content in real time. DESCRIPTION OF RELATED ART In the current approach user records an audio video clip using some handheld device like camcorder or mobile handset having video recording facility. He separately records his commentary in a new Audio file. He then needs to upload the recorded files to PC if editing software is not available in the handheld device. Using editing software he then merges two files or applies editing effects like audio mixing.Sometiems he removes original audio track in recorded video & then replaces it with recorded commentary using editing software. But this is very difficult & time consuming process. Also creatation of new media content depends on the editing features provided by editing software. If user wants to insert commentary in the original video at different places or not in continous form then user needs to record multiple audio files of proper duration & also without watching video of the original file. Inserting such multiple audio files into desired video at various point of time is very difficult using editing software. Currently there is no single application in any device available that allows user to perform recording & playback together in real time to form a new media content. Currently there is no single application available that allows user to add/mix live audio during playback to form a new media content in one device. In the current approach user needs to have multiple devices & software to add/mix live audio in already existing media content. In most of the devices or software it is not possible to create a media content similar to real time effect as they do not work in real time therefore further editing, processing is required. User need to spend more time, money in order to get such real time effect Setting the loudness level (Audio Signal level) before mixing for each audio is very important in sound mixing to get the best result out of sound mixing. With the help this setting user can decide which sound should be played as background & which sound should be played prominently in the output file. But many times user does not know the exact loudness level of the audio signal in the recorded file or live audio & does not have any control to change the loudness level of the any of the audio before mixing. Therefore user does not get the desired effects after mixing & experiences either very low level of one sound or very high level of other sound or background becomes prominent compare to main audio etc. Currently there is no single application in any device available that allows user to perform recording & playback together in real time to form a new media content. There is no single application available that allows user to add/mix live audio during playback to form a new media content in one device. In the current approach user needs to have multiple devices & software to add/mix live audio in already existing media content. In most of the devices or software it is not possible to create a media content similar to real time effect as they do not work in real time therefore further editing, processing is required. The user needs to spend more time, money in order to get such real time effect. Setting the loudness level (Audio Signal level) before mixing for each audio is very important in sound mixing to get the best result out of sound mixing. With the help this setting user can decide which sound should be played as background & which sound should be played prominently in the output file. But many times user does not know the exact loudness level of the audio signal in the recorded file or live audio & does not have any control to change the loudness level of the any of the audio before mixing. Therefore user does not get the desired effects after mixing & experiences either very low level of one sound or very high level of other sound or background becomes prominent compare to main audio etc. SUMMARY OF THE INVENTION The invention discloses a new use case of creating a new media content in real time by allowing user to perform playback & recording at a time which enables him to record live audio along with playback of recorded media file. The live audio can be inserted/enabled at any point of time during a playback of original file. The live audio when enabled during a playback will either replace or mix with original audio from the same instant in real time during playback. Live audio can be disabled at any point of time during a playback of original file. The live audio when disabled during a playback, application will stop replacing/mixing with original audio from the same instant in real time during playback. The invention discloses a new use case that allows the user to compare original audio with live audio in synchronization with original video. It further discloses a new use case to create new audio video content by combining original audio/video and live audio in real time that can be shared through MMS. The invention also discloses a new use case to create new audio video content by combining original video and mixed audio (live and original) in real time that can be shared through MMS. The invention further discloses a new use case which is "Audio Karaoke with original video" i.e adding live audio to already existing original video as original video provides a reference to live audio. It discloses a new use case to watch and listen comparison of original audio and live audio using recorded media file. User can dynamically view & change the loudness level or signal level of original audio & live audio dynamically before mixing to get the best audio mixing effect Accordingly this invention explains a method for creating new media content in real time using a mobile device comprising the step of: providing the mobile device with an audio processing unit; receiving live audio signal by the processing unit using a microphone; receiving pre-recorded audio-video signal from an input media file reader using an audio decoder; performing editing function with pre-recorded audio signal by the processing unit; and encoding the edited audio signal using an audio encoder and storing the said signal in a media file writer along with its corresponding video signal. Accordingly this invention also explains a system for creating new media content in real time in a mobile device comprising: an audio processing unit to process the audio signals and editing functions; a microphone for receiving live audio signal; an audio decoder to decode pre-recorded audio-video signal from an input media file reader; encoder for encoding the edited audio signal ;and memory for storing the encoded signal in a media file writer along with its corresponding video signal. These and other objects, features and advantages of the present invention will become more readily apparent from the detailed description taken in conjunction with the drawings and the claims. BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWING Figure 1 illustrates the components involved in the present invention. DETAILED DESCRIPTION OF THE INVENTION The preferred embodiments of the present invention will now be explained with reference to the accompanying drawings. It should be understood however that the disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. The following description and drawings are not to be construed as limiting the invention and numerous specific details are described to provide a thorough understanding of the present invention, as the basis for the claims and as a basis for teaching one skilled in the art how to make and/or use the invention. However in certain instances, well-known or conventional details are not described in order not to unnecessarily obscure the present invention in detail. The following are needed to implement the invention: • A Mobile Terminal with audio recording facility • Codecs supported by mobile phone media file format (3gp/3g2/mp4) • Mobile phone media file format reader & writer (3gp/3g2/mp4) • Audio mixing algorithm to mix audio samples from different streams in real time • Display device The following process happens in parallel and in real time 1. Input media file reading for audio and video 2. Decoding of audio and video streams from input media file 3. Play - out of decoded audio and display of decoded video from input media file 4. Capturing live audio 5. Calculating the loudness level of original audio & live audio, scaling up/down of the audio loudness level before mixing 6. Mixing of live audio and audio decoded from input media file 7. Encoding of live audio from microphone or mixed audio 8. Writing encoded audio and original encoded video from input file in a new media file. The application provides following features to the mobile user • Case-1 user can play both audio video of mobile phone media format (format 3gp/3g2/mp4) and record live audio captured by microphone at a time and can create a new audio video content including original audio- video and live audio. The original audio and live audio is mixed in real time and recorded along with original video in a new media file. The same operation is possible for audio only file also. • Case-2 user can play both audio and video of mobile phone media format (format 3gp/3g2/mp4) and record live audio captured by microphone at a time and can create a new audio video content including original audio, video and live audio. The original audio and live audio is not mixed in real time but original audio will be replaced by live audio in real time whenever inserted during a playback. Thus recorded file will have original video, original audio & live audio in a new media file. The same operation is possible for audio only file also. • Case-3 user can play only videos of audio/video file (format 3gp/3g2/mp4) and record live audio captured by microphone at a time & can create a new audio video content including original video and live audio. Thus recorded file will have original video only and live audio in a new media file. • Case-4 user can play video only mobile phone media format (format 3gp/3g2/mp4) and record live audio captured by microphone at a time and can create a new audio video content including original video and live audio. Thus recorded file will have original video live audio in a new media file. • In case-1, 2, 3, 4 live audio can be inserted/enabled at any point of time during a playback of original file. The live audio when enabled during a playback will replace/mix with original audio from the same instant in real time during playback. • Setting the loudness level (Audio Signal level) before mixing for each audio is very important in sound mixing to get the best result out of sound mixing. With the help this setting user can decide which sound should be played as background & which sound should be played prominently in the output file. But many times user does not know the exact loudness level of the audio signal in the recorded file or live audio & does not have any control to change the loudness level of the any of the audio before mixing. Therefore user does not get the desired effects after mixing & experiences either very low level of one sound or very high level of other sound or background becomes prominent compare to main audio etc. The "Smart Mixing " option helps user to solve this problem. With the help of this option user can experience & watch the loudness level of audio of original file during playback. He can also change the loudness level during the playback of original audio. Depending on the user requirement for the loudness level of original audio user can set & listen/experience the loudness level of original audio. Similarly user can set loudness level for inserting live audio as per his requirement. Based on the user input loudness level for original audio is scaled to the loudness level specified by the user dynamically before mixing. Similarly loudness level of live audio is also scaled to the level specified by the user dynamically before mixing. After scaling up/down the loudness level of both audio the signals will be added or mixed in real time. Due to this user gets the desired effect in the output file. • In case -1,2,3,4 live audio can be disabled at any point of time during a playback of original file. The live audio when disabled during a playback, the application will stop replacing/mixing with original audio from the same instant in real time during playback. • In case -1,2,3,4 due to dynamic enabling & disabling of live audio it is possible to replace/mix complete new audio. The insertion of live audio can be activated at the start of playback. • In case -1,2,3,4 due to dynamic enabling & disabling of live audio it is possible to replace/mix live audio one time or multiple times at any time whenever required during the playback of original file. Therefore it is possible to replace/mix part of original audio with live audio anytime anywhere during a playback of original file. • In case -1,2,3,4 it is possible to stop playback/recording anytime during playback of original file. With the help of this facility user can mix/replace original audio with live audio & can create a new file of required duration & size. • It allows user to do playback & recording simultaneously to compare original audio content & live audio content. • It allows user to record a real time comparison of live audio & original audio along with original video in media file format (3gp/3g2/mp4) • It provides a real time editing of audio by allowing user to mix original audio & live audio any time during a playback of original clip. • It provides a real time editing of audio by allowing user to replace a part of original audio & insert live audio any time during a playback of original clip. • The application works in real time & maintains the synchronization of audio & video during playback & recording. • In case of 1, 2, 3, 4 it is possible to add silence at any point of time during a playback of original file. The silence during a playback will replace/mix with original audio from the same instant in real time during playback. • Using above feature it is easy to remove unwanted audio portion in original file. Unwanted audio can be replaced with silence or with live audio during a playback of original file. Untouched audio portion remains as it is along with original video. • As the original audio/video clip is played, it is easy to watch video and add/insert audio at appropriate time/place wherever necessary. As user controls enabling/disabling of live audio during playback, it is easy to add comments/commentary any time anywhere in the original clip during a playback. • It can be used to add commentary, meaningful information in video only or audio/video clip • It can be used to remove unwanted portion with the silence. • It can be used to add some meaningful information in place of unwanted audio portion. • Information can be added at appropriate place by enabling /disabling live audio when user is watching video. • It can be used for Dialogue practicing & learning acting • It can be used to create funny videos, interesting videos • It can be used to highlight major events in video file by inserting comments at appropriate place • It can be used to create interactive videos • As recorded file is being in mobile phone media file format (3gp/3g2/mp4) it can be shared with other users through MMS or IMS applications. • There are some desktop audio-video editing application available but none of them provides a facility to play original clip, record live audio at a time and mixing of media streams to create a new mobile phone media file format (3gp/3g2/mp4). Some handheld devices are available for recording but complete functionality as explained above is not available in one device & user need to use multiple devices & tools to create such mobile phone media file format (3gp/3g2/mp4) which consumes time, cost & also multimedia knowledge to handle the tools is required. • The application reads two audio at a time one from input file and another from microphone and mixes two audio to form a single audio. The mixed audio and original video is written together in a new media file in mobile phone Original input audio - video file is played in synchronized form and provides a real time reference to record live audio. The video from input file and live audio is synchronized in real time and written to a file. • When application is launched it starts input output devices & major processing blocks like mobile phone media file reader, mobile phone media file writer, audio & video codecs & input/output devices like sound device, display device etc • Application reads the audio & video streams using mobile phone media file format reader block & decodes audio video bit-streams with the help of audio & video decoder. The video data decoded by decoder is displayed in Display screen & audio data decoded by audio decoder is played on speaker. The decoding is done by frame by frame. In other words application starts local playback of selected file. • If the live audio option is enabled (Insert live audio) by user during a playback then application also captures live audio data from microphone. Depending on the audio mixing option specified by user audio data from microphone is either mixed with the decoded audio data of input file & given to audio encoder or without mixing given to audio encoder depending on the mixing option set by the user. The video playout remain undisturbed during live audio insertion. Therefore user can insert live audio & can replace/mix with original audio while watching the video playback of the file. • User can disable live audio option at any point of time during playback of i the file as per his wish. • User can enable & disable live audio option multiple times during a playback of the file. • With the help of Smart Mixing option user can experience & watch the loudness level of audio of original file during playback. He can also change the loudness level during the playback of original audio. Depending on the user requirement for the loudness level of original audio user can set & listen/experience the loudness level of original audio. Similarly user can set loudness level for inserting live audio as per his requirement. Based on the user input loudness level for original audio is scaled to the loudness level specified by the user dynamically before mixing. Similarly loudness level of live audio is also scaled to the level specified by the user dynamically before mixing. After scaling up/down the loudness level of both audio the signals will be added or mixed in real time. Due to this user gets the desired effect in the output file. • Calculating the loudness level of original audio & live audio, scaling up/down of the audio before mixing & mixing of audio, inserting silence is performed by audio processing unit as shown in diagram. • mobile phone media file format writer receives encoded video data from input file & encoded audio data from audio encoder & writes to a new file in mobile phone media file format (3gp/3g2/mp4) • Application stops all devices & processing blocks when it detects an end of input mobile phone media file or if user stops application. The new recorded file is ready for playback when all devices & processing blocks are closed. Operation Details • User selects a media file • "Talk with me" application starts playing original media file • The application provides an option to enable/disable mixing of original audio and live audio during and before playback. If mixing is enabled application will mix live audio and original audio in real time otherwise live audio will replace original audio. • The user can watch playback of original media file and can enable live audio any time during playback. During a playback user can enable or disable live audio whenever he wants. If live audio is enabled then original audio will be replaced/mixed with live audio during that duration of playback. The live audio replace or mixing with original audio during that duration depends on audio mixing option set by user. If live audio is disabled then original audio will not be replaced/mixed during disabled time. • User can pause & resume playback of the original file at any time. This helps user to set various setting dynamically without affecting the playback & insertion of live audio. • With the help of Smart Mixing option user can experience & watch the loudness level of audio of original file during playback. He can also change the loudness level during the playback of original audio. Depending on the user requirement for the loudness level of original audio user can set & listen/experience the loudness level of original audio. Similarly user can set loudness level for inserting live audio as per his requirement. Based on the user input loudness level for original audio is scaled to the loudness level specified by the user dynamically before mixing. Similarly loudness level of live audio is also scaled to the level specified by the user dynamically before mixing. After scaling up/down the loudness level of both audio the signals will be added or mixed in real time. Due to this user gets the desired effect in the output file. • User can also insert silence anytime anywhere during a playback of a clip. • User can stop application anytime before the end of clip. This allows user to create a new media content of smaller size/duration. • If audio part is song or dialogue, user/third person can speak and can compare with original audio when media file created by application is played in player. • When application is launched it starts input output devices and major processing blocks like mobile phone media file reader, mobile phone media file writer, audio and video codecs and input/output devices like sound device, display device etc. • Application reads the audio and video streams using mobile phone media file format reader block and decodes audio video bit-streams with the help of audio and video decoder. The video data decoded by decoder is displayed in Display screen and audio data decoded by audio decoder is played on speaker. The decoding is done frame by frame. In other words application starts local playback of selected file in frame by frame mode. • If the live audio is enabled during a playback then application also captures live audio data from microphone. Depending on the audio mixing option specified by user audio data from microphone is either mixed with the decoded audio data of input file and given to audio encoder or without mixing given to audio encoder depending on the mixing option set by the user. • Mobile phone media file format writer receives encoded video data from input file and encoded audio data from audio encoder and writes to a new file in mobile phone media file format (3gp/3g2/mp4) • Application stops all devices & processing blocks when it detects an end of input mobile phone media file or if user stops application. • The new recorded file is ready for playback when all devices & processing blocks are closed. While the foregoing description assumed a particular architecture it will be appreciated that the present invention may be used in numerous architectures and in numerous other examples. The major advantages of the invented method over the existing art are that it allows usage of configured personality data while configuring and updating a personality. The foregoing description of illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed herein. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the present invention, as those skilled in the relevant art will recognize and appreciate. As indicated, these modifications may be made to the present invention in light of the foregoing description of illustrated embodiments of the present invention and are to be included within the spirit and scope of the present invention. GLOSSARY OF TERMS AND DEFINITIONS THEREOF 3gp 3GPP standard mobile phone media file format 3g2 3GPP2 standard mobile phone media file format mp4 MPEG-4 standard mobile phone media file format PC Personal Computer (Desktop Computer) MMS Multimedia messaging system IMS IP Multimedia Subsystem UE User Equipment We claim: 1. A method for creating new media content in real time using a mobile device comprising the step of: providing the mobile device with an audio processing unit; receiving live audio signal by the processing unit using a microphone; receiving pre-recorded audio-video signal from an input media file reader using an audio decoder; performing editing function with pre-recorded audio signal & live audio signal by the processing unit; and encoding the edited audio signal using an audio encoder and storing the said signal in a media file writer along with its corresponding video signal. 2. The method according to claim 1 wherein the audio processing unit performs editing function such as mixing, replacing or inserting live audio with pre¬recorded audio signal. 3. The method as claimed in claim 1 wherein the audio processing unit is adapted to inserting blank signal within audio signal based on user input. 4. The method according to claim 2 wherein if mixing is enabled a mobile device application mix live audio and original audio in real time if not a live audio will replace original audio. 5. The method according to claim 2 wherein if a live audio is enabled then an original audio will be replaced/mixed with live audio during that duration depends for the audio mixing option. 6. The method according to claim 2 wherein the loudness level for the original and the live audio can be configured. 7. The method according to claim 2 wherein depending on the audio mixing option audio data from microphone is either mixed with the decoded audio data of input file and given to audio encoder or without mixing given to audio encoder depending on the mixing option set. 8. The method according to claim 2 wherein the media file writer receives encoded video data from input file and encoded audio data from audio encoder and writes to a new file in mobile phone media file format. 9. The method according to claim 1 wherein live comments can be added, inserted or mixed single or multiple times during playback of audio-video file, video only file & for audio only file. 10. The method according to claim 1 wherein the original audio can be compared with live audio in synchronization with original video. 11. A system for creating new media content in real time in a mobile device comprising: an audio processing unit to process the audio signals and editing functions; a microphone for receiving live audio signal; an audio decoder to decode pre-recorded audio-video signal from an input media file reader; encoder for encoding the edited audio signal ;and memory for storing the encoded signal in a media file writer along with its corresponding video signal. 12. The method according to claim 2 wherein user can check/observe the loudness level of the original and the live audio in real time 13. A method for creating new media content in real time using a mobile device substantially described particularly with reference to the accompanying drawings 14. A system for creating new media content in real time in a mobile device substantially described particularly with reference to the accompanying drawings.

Full Text

FIELD OF INVENTION
The present invention, in general, relates to the field of mobile communication. The invention describes Software Desktop application, feature phone (mobile phone) and smart phone. This invention relates to a new use case for desktop, mobile phone with audio record facility. More particularly, this invention relates to a system and method for creating new media content in real time.
DESCRIPTION OF RELATED ART
In the current approach user records an audio video clip using some handheld device like camcorder or mobile handset having video recording facility. He separately records his commentary in a new Audio file. He then needs to upload the recorded files to PC if editing software is not available in the handheld device. Using editing software he then merges two files or applies editing effects like audio mixing.Sometiems he removes original audio track in recorded video & then replaces it with recorded commentary using editing software. But this is very difficult & time consuming process. Also creatation of new media content depends on the editing features provided by editing software. If user wants to insert commentary in the original video at different places or not in continous form then user needs to record multiple audio files of proper duration & also without watching video of the original file. Inserting such multiple audio files into desired video at various point of time is very difficult using editing software. Currently there is no single application in any device available that allows user to perform recording & playback together in real time to form a new media content.
Currently there is no single application available that allows user to add/mix live audio during playback to form a new media content in one device.
In the current approach user needs to have multiple devices & software to add/mix live audio in already existing media content.
In most of the devices or software it is not possible to create a media content similar to real time effect as they do not work in real time therefore further editing, processing is required.
User need to spend more time, money in order to get such real time effect Setting the loudness level (Audio Signal level) before mixing for each audio is very important in sound mixing to get the best result out of sound mixing. With the help this setting user can decide which sound should be played as background & which sound should be played prominently in the output file. But many times user does not know the exact loudness level of the audio signal in the recorded file or live audio & does not have any control to change the loudness level of the any of the audio before mixing. Therefore user does not get the desired effects after mixing & experiences either very low level of one sound or very high level of other sound or background becomes prominent compare to main audio etc.
Currently there is no single application in any device available that allows user to perform recording & playback together in real time to form a new media content. There is no single application available that allows user to add/mix live audio during playback to form a new media content in one device. In the current approach user needs to have multiple devices & software to add/mix live audio in already existing media content. In most of the devices or software it is not possible to create a media content similar to real time effect as they do not work in real time therefore further editing, processing is required. The user needs to spend more time, money in order to get such real time effect. Setting the loudness level (Audio Signal level) before mixing for each audio is very important in sound mixing to get the best result out of sound mixing. With the help this setting user can decide which sound should be played as background & which sound should be played prominently in the output file. But many times user does not know the exact loudness level of the audio signal in the recorded file or live audio & does not have any control to change the loudness level of the any of the audio before mixing. Therefore user does not get the desired effects after mixing & experiences either very low level of one sound or very high level of other sound or background becomes prominent compare to main audio etc.
SUMMARY OF THE INVENTION
The invention discloses a new use case of creating a new media content in real time by allowing user to perform playback & recording at a time which enables him to record live audio along with playback of recorded media file. The live audio can be inserted/enabled at any point of time during a playback of original file. The live audio when enabled during a playback will either replace or mix with original audio from the same instant in real time during playback. Live audio can be disabled at any point of time during a playback of original file. The live audio when disabled during a playback, application will stop replacing/mixing with original audio from the same instant in real time during playback. The invention discloses a new use case that allows the user to compare original audio with live audio in synchronization with original video. It further discloses a new use case to create new audio video content by combining original audio/video and live audio in real time that can be shared through MMS. The invention also discloses a new use case to create new audio video content by combining original video and mixed audio (live and original) in real time that can be shared through MMS. The invention further discloses a new use case which is "Audio Karaoke with original video" i.e adding live audio to already existing original video as original video provides a reference to live audio. It discloses a new use case to watch and listen comparison of original audio and live audio using recorded media file. User can dynamically view & change the loudness level or signal level of original audio & live audio dynamically before mixing to get the best audio mixing effect
Accordingly this invention explains a method for creating new media content in real time using a mobile device comprising the step of:
providing the mobile device with an audio processing unit;
receiving live audio signal by the processing unit using a microphone;
receiving pre-recorded audio-video signal from an input media file reader using an audio decoder;
performing editing function with pre-recorded audio signal by the processing unit; and
encoding the edited audio signal using an audio encoder and storing the said signal in a media file writer along with its corresponding video signal.
Accordingly this invention also explains a system for creating new media content in real time in a mobile device comprising:
an audio processing unit to process the audio signals and editing functions;
a microphone for receiving live audio signal;
an audio decoder to decode pre-recorded audio-video signal from an input media file reader;
encoder for encoding the edited audio signal ;and
memory for storing the encoded signal in a media file writer along with its corresponding video signal.
These and other objects, features and advantages of the present invention will become more readily apparent from the detailed description taken in conjunction
with the drawings and the claims.
BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWING Figure 1 illustrates the components involved in the present invention. DETAILED DESCRIPTION OF THE INVENTION
The preferred embodiments of the present invention will now be explained with reference to the accompanying drawings. It should be understood however that the disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. The following description and drawings are not to be construed as limiting the invention and numerous specific details are described to provide a thorough understanding of the present invention, as the basis for the claims and as a basis for teaching one skilled in the art how to make and/or use the invention. However in certain instances, well-known or conventional details are not described in order not to unnecessarily obscure the present invention in detail.
The following are needed to implement the invention:
• A Mobile Terminal with audio recording facility
• Codecs supported by mobile phone media file format (3gp/3g2/mp4)
• Mobile phone media file format reader & writer (3gp/3g2/mp4)
• Audio mixing algorithm to mix audio samples from different streams in real time
• Display device
The following process happens in parallel and in real time
1. Input media file reading for audio and video
2. Decoding of audio and video streams from input media file
3. Play - out of decoded audio and display of decoded video from input media file
4. Capturing live audio
5. Calculating the loudness level of original audio & live audio, scaling up/down of the audio loudness level before mixing
6. Mixing of live audio and audio decoded from input media file
7. Encoding of live audio from microphone or mixed audio
8. Writing encoded audio and original encoded video from input file in a new media file.
The application provides following features to the mobile user
• Case-1 user can play both audio video of mobile phone media format (format 3gp/3g2/mp4) and record live audio captured by microphone at a time and can create a new audio video content including original audio- video and live audio. The original audio and live audio is mixed in real time and recorded along with original video in a new media file. The same operation is possible for audio only file also.
• Case-2 user can play both audio and video of mobile phone media format (format 3gp/3g2/mp4) and record live audio captured by microphone at a time and can create a new audio video content including original audio, video and live audio. The original audio and live audio is not mixed in real time but original audio will be replaced by live audio in real time whenever inserted during a playback. Thus recorded file will have original video, original audio & live audio in a new media file. The same operation is possible for audio only file also.
• Case-3 user can play only videos of audio/video file (format 3gp/3g2/mp4) and record live audio captured by microphone at a time & can create a new audio video content including original video and live audio. Thus recorded file will have original video only and live audio in a new media file.
• Case-4 user can play video only mobile phone media format (format 3gp/3g2/mp4) and record live audio captured by microphone at a time and can create a new audio video content including original video and live audio. Thus recorded file will have original video live audio in a new media file.
• In case-1, 2, 3, 4 live audio can be inserted/enabled at any point of time during a playback of original file. The live audio when enabled during a
playback will replace/mix with original audio from the same instant in real time during playback.
• Setting the loudness level (Audio Signal level) before mixing for each audio is very important in sound mixing to get the best result out of sound mixing. With the help this setting user can decide which sound should be played as background & which sound should be played prominently in the output file. But many times user does not know the exact loudness level of the audio signal in the recorded file or live audio & does not have any control to change the loudness level of the any of the audio before mixing. Therefore user does not get the desired effects after mixing & experiences either very low level of one sound or very high level of other sound or background becomes prominent compare to main audio etc. The "Smart Mixing " option helps user to solve this problem. With the help of this option user can experience & watch the loudness level of audio of original file during playback. He can also change the loudness level during the playback of original audio. Depending on the user requirement for the loudness level of original audio user can set & listen/experience the loudness level of original audio. Similarly user can set loudness level for inserting live audio as per his requirement. Based on the user input loudness level for original audio is scaled to the loudness level specified by the user dynamically before mixing. Similarly loudness level of live audio is also scaled to the level specified by the user dynamically before mixing. After scaling up/down the loudness level of both audio the signals will be added or mixed in real time. Due to this user gets the desired effect in the output file.
• In case -1,2,3,4 live audio can be disabled at any point of time during a playback of original file.
The live audio when disabled during a playback, the application will stop replacing/mixing with original audio from the same instant in real time during playback.
• In case -1,2,3,4 due to dynamic enabling & disabling of live audio it is possible to replace/mix complete new audio. The insertion of live audio can be activated at the start of playback.
• In case -1,2,3,4 due to dynamic enabling & disabling of live audio it is possible to replace/mix live audio one time or multiple times at any time whenever required during the playback of original file. Therefore it is possible to replace/mix part of original audio with live audio anytime anywhere during a playback of original file.
• In case -1,2,3,4 it is possible to stop playback/recording anytime during playback of original file. With the help of this facility user can mix/replace original audio with live audio & can create a new file of required duration & size.
• It allows user to do playback & recording simultaneously to compare original audio content & live audio content.
• It allows user to record a real time comparison of live audio & original audio along with original video in media file format (3gp/3g2/mp4)
• It provides a real time editing of audio by allowing user to mix original audio & live audio any time during a playback of original clip.
• It provides a real time editing of audio by allowing user to replace a part of original audio & insert live audio any time during a playback of original clip.
• The application works in real time & maintains the synchronization of audio & video during playback & recording.
• In case of 1, 2, 3, 4 it is possible to add silence at any point of time during a playback of original file. The silence during a playback will replace/mix with original audio from the same instant in real time during playback.
• Using above feature it is easy to remove unwanted audio portion in original file. Unwanted audio can be replaced with silence or with live audio during a playback of original file. Untouched audio portion remains as it is along with original video.
• As the original audio/video clip is played, it is easy to watch video and add/insert audio at appropriate time/place wherever necessary. As user controls enabling/disabling of live audio during playback, it is easy to add comments/commentary any time anywhere in the original clip during a playback.
• It can be used to add commentary, meaningful information in video only or audio/video clip
• It can be used to remove unwanted portion with the silence.
• It can be used to add some meaningful information in place of unwanted audio portion.
• Information can be added at appropriate place by enabling /disabling live audio when user is watching video.
• It can be used for Dialogue practicing & learning acting
• It can be used to create funny videos, interesting videos
• It can be used to highlight major events in video file by inserting comments at appropriate place
• It can be used to create interactive videos
• As recorded file is being in mobile phone media file format (3gp/3g2/mp4) it can be shared with other users through MMS or IMS applications.
• There are some desktop audio-video editing application available but none of them provides a facility to play original clip, record live audio at a time and mixing of media streams to create a new mobile phone media file format (3gp/3g2/mp4). Some handheld devices are available for recording but complete functionality as explained above is not available in one device & user need to use multiple devices & tools to create such mobile phone media file format (3gp/3g2/mp4) which consumes time, cost & also multimedia knowledge to handle the tools is required.
• The application reads two audio at a time one from input file and another from microphone and mixes two audio to form a single audio. The mixed audio and original video is written together in a new media file in mobile phone Original input audio - video file is played in synchronized form and provides a real time reference to record live audio. The video from input file and live audio is synchronized in real time and written to a file.
• When application is launched it starts input output devices & major processing blocks like mobile phone media file reader, mobile phone media file writer, audio & video codecs & input/output devices like sound device, display device etc
• Application reads the audio & video streams using mobile phone media file format reader block & decodes audio video bit-streams with the help of audio & video decoder. The video data decoded by decoder is displayed in Display screen & audio data decoded by audio decoder is played on speaker. The decoding is done by frame by frame. In other words application starts local playback of selected file.
• If the live audio option is enabled (Insert live audio) by user during a playback then application also captures live audio data from microphone. Depending on the audio mixing option specified by user audio data from microphone is either mixed with the decoded audio data of input file & given to audio encoder or without mixing given to audio encoder depending on the mixing option set by the user. The video playout remain undisturbed during live audio insertion. Therefore user can insert live audio & can replace/mix with original audio while watching the video playback of the file.
• User can disable live audio option at any point of time during playback of
i
the file as per his wish.
• User can enable & disable live audio option multiple times during a playback of the file.
• With the help of Smart Mixing option user can experience & watch the loudness level of audio of original file during playback. He can also change the loudness level during the playback of original audio. Depending on the user requirement for the loudness level of original audio user can set & listen/experience the loudness level of original audio. Similarly user can set loudness level for inserting live audio as per his requirement. Based on the user input loudness level for original audio is scaled to the loudness level specified by the user dynamically before
mixing. Similarly loudness level of live audio is also scaled to the level specified by the user dynamically before mixing. After scaling up/down the loudness level of both audio the signals will be added or mixed in real time. Due to this user gets the desired effect in the output file.
• Calculating the loudness level of original audio & live audio, scaling up/down of the audio before mixing & mixing of audio, inserting silence is performed by audio processing unit as shown in diagram.
• mobile phone media file format writer receives encoded video data from input file & encoded audio data from audio encoder & writes to a new file in mobile phone media file format (3gp/3g2/mp4)
• Application stops all devices & processing blocks when it detects an end of input mobile phone media file or if user stops application.
The new recorded file is ready for playback when all devices & processing blocks are closed.
Operation Details
• User selects a media file
• "Talk with me" application starts playing original media file
• The application provides an option to enable/disable mixing of original audio and live audio during and before playback. If mixing is enabled application will mix live audio and original audio in real time otherwise live audio will replace original audio.
• The user can watch playback of original media file and can enable live audio any time during playback. During a playback user can enable or disable live audio whenever he wants. If live audio is enabled then original audio will be replaced/mixed with live audio during that duration of playback. The live audio replace or mixing with original audio during that duration depends on audio mixing option set by user. If live audio is disabled then original audio will not be replaced/mixed during disabled time.
• User can pause & resume playback of the original file at any time. This helps user to set various setting dynamically without affecting the playback & insertion of live audio.
• With the help of Smart Mixing option user can experience & watch the loudness level of audio of original file during playback. He can also change the loudness level during the playback of original audio. Depending on the user requirement for the loudness level of original audio user can set & listen/experience the loudness level of original audio. Similarly user can set loudness level for inserting live audio as per his requirement. Based on the user input loudness level for original audio is scaled to the loudness level specified by the user dynamically before mixing. Similarly loudness level of live audio is also scaled to the level specified by the user dynamically before mixing. After scaling up/down the loudness level of both audio the signals will be added or mixed in real time. Due to this user gets the desired effect in the output file.
• User can also insert silence anytime anywhere during a playback of a clip.
• User can stop application anytime before the end of clip. This allows user to create a new media content of smaller size/duration.
• If audio part is song or dialogue, user/third person can speak and can compare with original audio when media file created by application is played in player.
• When application is launched it starts input output devices and major processing blocks like mobile phone media file reader, mobile phone media file writer, audio and video codecs and input/output devices like sound device, display device etc.
• Application reads the audio and video streams using mobile phone media file format reader block and decodes audio video bit-streams with the help of audio and video decoder. The video data decoded by decoder is displayed in Display screen and audio data decoded by audio decoder is played on speaker. The decoding is done frame by frame. In other words application starts local playback of selected file in frame by frame mode.
• If the live audio is enabled during a playback then application also captures live audio data from microphone. Depending on the audio mixing option specified by user audio data from microphone is either mixed with the decoded audio data of input file and given to audio encoder or without mixing given to audio encoder depending on the mixing option set by the user.
• Mobile phone media file format writer receives encoded video data from input file and encoded audio data from audio encoder and writes to a new file in mobile phone media file format (3gp/3g2/mp4)
• Application stops all devices & processing blocks when it detects an end of input mobile phone media file or if user stops application.
• The new recorded file is ready for playback when all devices & processing blocks are closed.
While the foregoing description assumed a particular architecture it will be appreciated that the present invention may be used in numerous architectures and in numerous other examples. The major advantages of the invented method over the existing art are that it allows usage of configured personality data while configuring and updating a personality.
The foregoing description of illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed herein. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the present invention, as those skilled in the relevant art will recognize and appreciate. As indicated, these modifications may be made to the present invention in light of the foregoing description of illustrated embodiments of the present invention and are to be included within the spirit and scope of the present invention.
GLOSSARY OF TERMS AND DEFINITIONS THEREOF
3gp 3GPP standard mobile phone media file format
3g2 3GPP2 standard mobile phone media file format
mp4 MPEG-4 standard mobile phone media file format
PC Personal Computer (Desktop Computer)
MMS Multimedia messaging system
IMS IP Multimedia Subsystem
UE User Equipment

We claim:
1. A method for creating new media content in real time using a mobile device comprising the step of:
providing the mobile device with an audio processing unit;
receiving live audio signal by the processing unit using a microphone;
receiving pre-recorded audio-video signal from an input media file reader using an audio decoder;
performing editing function with pre-recorded audio signal & live audio signal by the processing unit; and
encoding the edited audio signal using an audio encoder and storing the said signal in a media file writer along with its corresponding video signal.
2. The method according to claim 1 wherein the audio processing unit performs editing function such as mixing, replacing or inserting live audio with pre¬recorded audio signal.
3. The method as claimed in claim 1 wherein the audio processing unit is adapted to inserting blank signal within audio signal based on user input.
4. The method according to claim 2 wherein if mixing is enabled a mobile device application mix live audio and original audio in real time if not a live audio will replace original audio.
5. The method according to claim 2 wherein if a live audio is enabled then an original audio will be replaced/mixed with live audio during that duration depends for the audio mixing option.
6. The method according to claim 2 wherein the loudness level for the original and the live audio can be configured.
7. The method according to claim 2 wherein depending on the audio mixing option audio data from microphone is either mixed with the decoded audio data of input file and given to audio encoder or without mixing given to audio encoder depending on the mixing option set.
8. The method according to claim 2 wherein the media file writer receives encoded video data from input file and encoded audio data from audio encoder and writes to a new file in mobile phone media file format.
9. The method according to claim 1 wherein live comments can be added, inserted or mixed single or multiple times during playback of audio-video file, video only file & for audio only file.
10. The method according to claim 1 wherein the original audio can be compared with live audio in synchronization with original video.
11. A system for creating new media content in real time in a mobile device comprising:
an audio processing unit to process the audio signals and editing functions;
a microphone for receiving live audio signal;
an audio decoder to decode pre-recorded audio-video signal from an input media file reader;
encoder for encoding the edited audio signal ;and
memory for storing the encoded signal in a media file writer along with its corresponding video signal.
12. The method according to claim 2 wherein user can check/observe the loudness level of the original and the live audio in real time
13. A method for creating new media content in real time using a mobile device substantially described particularly with reference to the accompanying drawings
14. A system for creating new media content in real time in a mobile device substantially described particularly with reference to the accompanying drawings.

Documents:

http://ipindiaonline.gov.in/patentsearch/GrantedSearch/viewdoc.aspx?id=53MQxnJTqk/KDCtFoN+dPw==&loc=egcICQiyoj82NGgGrC5ChA==

« Previous Patent

Next Patent »

Patent Number

271764

Indian Patent Application Number

1128/CHE/2006

PG Journal Number

10/2016

Publication Date

04-Mar-2016

Grant Date

03-Mar-2016

Date of Filing

30-Jun-2006

Name of Patentee

SAMSUNG R& D INSTITUTE INDIA BANGALORE PRIVATE LIMITED

Applicant Address

#2870 ORION BUILDING BAGMANE CONSTELLATION BUSINESS PARK OUTER RING ROAD DODDANEKUNDI CIRCLE MARATHAHALLI POST BANGALORE -560037

Inventors:

#	Inventor's Name	Inventor's Address
1	GIRISH KULKARNI,	BLOCK 'B' NO.66/1 , BAGMANE PARK , C V RAMAN NAGAR, BYRASANDRA , BANGALORE ,
2	MAHESH PATIL,	BLOCK 'B' NO.66/1 , BAGMANE PARK , C V RAMAN NAGAR, BYRASANDRA , BANGALORE ,560 093 ,

PCT International Classification Number

G06F 17/00

PCT International Application Number

N/A

PCT International Filing date

PCT Conventions:

#	PCT Application Number	Date of Convention	Priority Country
1			NA