|
|
Created:
7 years, 4 months ago by Raymond Toy (Google) Modified:
7 years, 4 months ago CC:
chromium-reviews, joi+watch-content_chromium.org, darin-cc_chromium.org, jam, feature-media-reviews_chromium.org Base URL:
https://chromium.googlesource.com/chromium/src.git@master Visibility:
Public. |
DescriptionFix slow playback of 1-channel MP3 and AAC files for WebAudio on Android
BUG=266006
If the MediaFormat indicates 1 channel for the file but the MediaCodec decoder changes the output format to 2 channels, we remove the data from the second channel before sending to the receiver. This makes the the transmitted data consistent with the expected number of channels.
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=216779
Patch Set 1 #Patch Set 2 : New implementation #
Total comments: 6
Patch Set 3 : #
Messages
Total messages: 18 (0 generated)
Please take a look. I'm not sure this is the best possible workaround, but I couldn't think of anything else to do.
I don't think this is appropriate fix. Shouldn't this be fixed in decoder side instead of audio_decoder_android? Fixing it at decoder side helps us to pass less data for reading/writing
On 2013/08/02 07:00:53, qinmin wrote: > I don't think this is appropriate fix. Shouldn't this be fixed in decoder side > instead of audio_decoder_android? Fixing it at decoder side helps us to pass > less data for reading/writing Yes, it could be done on the browser (decoder) side. But then the browser will have to buffer the entire decoded data so that it can make the decision. Currently, the browser just sends the decoded chunks to the renderer. I would prefer not to increase memory usage. The current memory usage is: renderer has two copies of the encoded data (one is the original and one for the shared memory buffer) and required destination bus for the decoded data. In addition, the renderer may also have to buffer the decoded data if the size is unknown. On the browser, we save a copy of the shared memory buffer into a file (to fix a different Android issue). There's a small buffer for holding decoded chunks. If we change this to do it on the browser side, the browser now gets a full copy of the decoded data. The decoded data is much larger than the encoded data, so we are using much more memory now.
On 2013/08/02 16:07:22, rtoy wrote: > On 2013/08/02 07:00:53, qinmin wrote: > > I don't think this is appropriate fix. Shouldn't this be fixed in decoder side > > instead of audio_decoder_android? Fixing it at decoder side helps us to pass > > less data for reading/writing > > Yes, it could be done on the browser (decoder) side. But then the browser will > have to buffer the entire decoded data so that it can make the decision. > Currently, the browser just sends the decoded chunks to the renderer. I would > prefer not to increase memory usage. > > The current memory usage is: renderer has two copies of the encoded data (one > is the original and one for the shared memory buffer) and required destination > bus for the decoded data. In addition, the renderer may also have to buffer the > decoded data if the size is unknown. > > On the browser, we save a copy of the shared memory buffer into a file (to fix a > different Android issue). There's a small buffer for holding decoded chunks. > > If we change this to do it on the browser side, the browser now gets a full copy > of the decoded data. The decoded data is much larger than the encoded data, so > we are using much more memory now. But why mediaCodec generates 2 channels instead of 1? I think there must be some configuration problems. And for the decoded chunks, can browser side just remove one channel before returning it to the renderer?
On 2013/08/02 16:12:04, qinmin wrote: > On 2013/08/02 16:07:22, rtoy wrote: > > On 2013/08/02 07:00:53, qinmin wrote: > > > I don't think this is appropriate fix. Shouldn't this be fixed in decoder > side > > > instead of audio_decoder_android? Fixing it at decoder side helps us to pass > > > less data for reading/writing > > > > Yes, it could be done on the browser (decoder) side. But then the browser > will > > have to buffer the entire decoded data so that it can make the decision. > > Currently, the browser just sends the decoded chunks to the renderer. I would > > prefer not to increase memory usage. > > > > The current memory usage is: renderer has two copies of the encoded data (one > > is the original and one for the shared memory buffer) and required destination > > bus for the decoded data. In addition, the renderer may also have to buffer > the > > decoded data if the size is unknown. > > > > On the browser, we save a copy of the shared memory buffer into a file (to fix > a > > different Android issue). There's a small buffer for holding decoded chunks. > > > > If we change this to do it on the browser side, the browser now gets a full > copy > > of the decoded data. The decoded data is much larger than the encoded data, > so > > we are using much more memory now. > > But why mediaCodec generates 2 channels instead of 1? I think there must be some > configuration problems. > And for the decoded chunks, can browser side just remove one channel before > returning it to the renderer? I believe it's a bug in the Android implementation. This doesn't happen on Nexus devices that I've tested. As for getting the browser to remove one channel, I don't know of any way to tell if the decoded chunk has two channels or one channel in it. This is why I check that the final decoded length is approximately twice the expected length.
On 2013/08/02 16:26:00, rtoy wrote: > On 2013/08/02 16:12:04, qinmin wrote: > > On 2013/08/02 16:07:22, rtoy wrote: > > > On 2013/08/02 07:00:53, qinmin wrote: > > > > I don't think this is appropriate fix. Shouldn't this be fixed in decoder > > side > > > > instead of audio_decoder_android? Fixing it at decoder side helps us to > pass > > > > less data for reading/writing > > > > > > Yes, it could be done on the browser (decoder) side. But then the browser > > will > > > have to buffer the entire decoded data so that it can make the decision. > > > Currently, the browser just sends the decoded chunks to the renderer. I > would > > > prefer not to increase memory usage. > > > > > > The current memory usage is: renderer has two copies of the encoded data > (one > > > is the original and one for the shared memory buffer) and required > destination > > > bus for the decoded data. In addition, the renderer may also have to buffer > > the > > > decoded data if the size is unknown. > > > > > > On the browser, we save a copy of the shared memory buffer into a file (to > fix > > a > > > different Android issue). There's a small buffer for holding decoded > chunks. > > > > > > If we change this to do it on the browser side, the browser now gets a full > > copy > > > of the decoded data. The decoded data is much larger than the encoded data, > > so > > > we are using much more memory now. > > > > But why mediaCodec generates 2 channels instead of 1? I think there must be > some > > configuration problems. > > And for the decoded chunks, can browser side just remove one channel before > > returning it to the renderer? > > I believe it's a bug in the Android implementation. This doesn't happen on > Nexus devices that I've tested. As for getting the browser to remove one > channel, I don't know of any way to tell if the decoded chunk has two channels > or one channel in it. This is why I check that the final decoded length is > approximately twice the expected length. I think we'd like to understand the source of the problem rather than hacing it in the renderer side. When calling MediaCodec, can you check what is the result from format.getInteger(MediaFormat.KEY_CHANNEL_COUNT); if the returned channel count is 1, can you check whether adding a call to MediaFormat.setInteger(MediaFormat.KEY_CHANNEL_MASK, AudioFormat.CHANNEL_OUT_MONO) helps.
On 2013/08/02 16:38:08, qinmin wrote: > On 2013/08/02 16:26:00, rtoy wrote: > > On 2013/08/02 16:12:04, qinmin wrote: > > > On 2013/08/02 16:07:22, rtoy wrote: > > > > On 2013/08/02 07:00:53, qinmin wrote: > > > > > I don't think this is appropriate fix. Shouldn't this be fixed in > decoder > > > side > > > > > instead of audio_decoder_android? Fixing it at decoder side helps us to > > pass > > > > > less data for reading/writing > > > > > > > > Yes, it could be done on the browser (decoder) side. But then the browser > > > will > > > > have to buffer the entire decoded data so that it can make the decision. > > > > Currently, the browser just sends the decoded chunks to the renderer. I > > would > > > > prefer not to increase memory usage. > > > > > > > > The current memory usage is: renderer has two copies of the encoded data > > (one > > > > is the original and one for the shared memory buffer) and required > > destination > > > > bus for the decoded data. In addition, the renderer may also have to > buffer > > > the > > > > decoded data if the size is unknown. > > > > > > > > On the browser, we save a copy of the shared memory buffer into a file (to > > fix > > > a > > > > different Android issue). There's a small buffer for holding decoded > > chunks. > > > > > > > > If we change this to do it on the browser side, the browser now gets a > full > > > copy > > > > of the decoded data. The decoded data is much larger than the encoded > data, > > > so > > > > we are using much more memory now. > > > > > > But why mediaCodec generates 2 channels instead of 1? I think there must be > > some > > > configuration problems. > > > And for the decoded chunks, can browser side just remove one channel before > > > returning it to the renderer? > > > > I believe it's a bug in the Android implementation. This doesn't happen on > > Nexus devices that I've tested. As for getting the browser to remove one > > channel, I don't know of any way to tell if the decoded chunk has two channels > > or one channel in it. This is why I check that the final decoded length is > > approximately twice the expected length. > > I think we'd like to understand the source of the problem rather than hacing it > in the renderer side. > When calling MediaCodec, can you check what is the result from > format.getInteger(MediaFormat.KEY_CHANNEL_COUNT); > if the returned channel count is 1, can you check whether adding a call to > MediaFormat.setInteger(MediaFormat.KEY_CHANNEL_MASK, > AudioFormat.CHANNEL_OUT_MONO) helps. KEY_CHANNEL_COUNT is definitely 1. I will try out the setInteger. This is a much better solution if it works.
On 2013/08/02 16:42:12, rtoy wrote: > On 2013/08/02 16:38:08, qinmin wrote: > > On 2013/08/02 16:26:00, rtoy wrote: > > > On 2013/08/02 16:12:04, qinmin wrote: > > > > On 2013/08/02 16:07:22, rtoy wrote: > > > > > On 2013/08/02 07:00:53, qinmin wrote: > > > > > > I don't think this is appropriate fix. Shouldn't this be fixed in > > decoder > > > > side > > > > > > instead of audio_decoder_android? Fixing it at decoder side helps us > to > > > pass > > > > > > less data for reading/writing > > > > > > > > > > Yes, it could be done on the browser (decoder) side. But then the > browser > > > > will > > > > > have to buffer the entire decoded data so that it can make the decision. > > > > > > Currently, the browser just sends the decoded chunks to the renderer. I > > > would > > > > > prefer not to increase memory usage. > > > > > > > > > > The current memory usage is: renderer has two copies of the encoded > data > > > (one > > > > > is the original and one for the shared memory buffer) and required > > > destination > > > > > bus for the decoded data. In addition, the renderer may also have to > > buffer > > > > the > > > > > decoded data if the size is unknown. > > > > > > > > > > On the browser, we save a copy of the shared memory buffer into a file > (to > > > fix > > > > a > > > > > different Android issue). There's a small buffer for holding decoded > > > chunks. > > > > > > > > > > If we change this to do it on the browser side, the browser now gets a > > full > > > > copy > > > > > of the decoded data. The decoded data is much larger than the encoded > > data, > > > > so > > > > > we are using much more memory now. > > > > > > > > But why mediaCodec generates 2 channels instead of 1? I think there must > be > > > some > > > > configuration problems. > > > > And for the decoded chunks, can browser side just remove one channel > before > > > > returning it to the renderer? > > > > > > I believe it's a bug in the Android implementation. This doesn't happen on > > > Nexus devices that I've tested. As for getting the browser to remove one > > > channel, I don't know of any way to tell if the decoded chunk has two > channels > > > or one channel in it. This is why I check that the final decoded length is > > > approximately twice the expected length. > > > > I think we'd like to understand the source of the problem rather than hacing > it > > in the renderer side. > > When calling MediaCodec, can you check what is the result from > > format.getInteger(MediaFormat.KEY_CHANNEL_COUNT); > > if the returned channel count is 1, can you check whether adding a call to > > MediaFormat.setInteger(MediaFormat.KEY_CHANNEL_MASK, > > AudioFormat.CHANNEL_OUT_MONO) helps. > > KEY_CHANNEL_COUNT is definitely 1. I will try out the setInteger. This is a > much better solution if it works. Setting the channel mask to mono doesn't help. :-(
On 2013/08/02 17:00:51, rtoy wrote: > On 2013/08/02 16:42:12, rtoy wrote: > > On 2013/08/02 16:38:08, qinmin wrote: > > > On 2013/08/02 16:26:00, rtoy wrote: > > > > On 2013/08/02 16:12:04, qinmin wrote: > > > > > On 2013/08/02 16:07:22, rtoy wrote: > > > > > > On 2013/08/02 07:00:53, qinmin wrote: > > > > > > > I don't think this is appropriate fix. Shouldn't this be fixed in > > > decoder > > > > > side > > > > > > > instead of audio_decoder_android? Fixing it at decoder side helps us > > to > > > > pass > > > > > > > less data for reading/writing > > > > > > > > > > > > Yes, it could be done on the browser (decoder) side. But then the > > browser > > > > > will > > > > > > have to buffer the entire decoded data so that it can make the > decision. > > > > > > > > Currently, the browser just sends the decoded chunks to the renderer. > I > > > > would > > > > > > prefer not to increase memory usage. > > > > > > > > > > > > The current memory usage is: renderer has two copies of the encoded > > data > > > > (one > > > > > > is the original and one for the shared memory buffer) and required > > > > destination > > > > > > bus for the decoded data. In addition, the renderer may also have to > > > buffer > > > > > the > > > > > > decoded data if the size is unknown. > > > > > > > > > > > > On the browser, we save a copy of the shared memory buffer into a file > > (to > > > > fix > > > > > a > > > > > > different Android issue). There's a small buffer for holding decoded > > > > chunks. > > > > > > > > > > > > If we change this to do it on the browser side, the browser now gets a > > > full > > > > > copy > > > > > > of the decoded data. The decoded data is much larger than the encoded > > > data, > > > > > so > > > > > > we are using much more memory now. > > > > > > > > > > But why mediaCodec generates 2 channels instead of 1? I think there must > > be > > > > some > > > > > configuration problems. > > > > > And for the decoded chunks, can browser side just remove one channel > > before > > > > > returning it to the renderer? > > > > > > > > I believe it's a bug in the Android implementation. This doesn't happen > on > > > > Nexus devices that I've tested. As for getting the browser to remove one > > > > channel, I don't know of any way to tell if the decoded chunk has two > > channels > > > > or one channel in it. This is why I check that the final decoded length > is > > > > approximately twice the expected length. > > > > > > I think we'd like to understand the source of the problem rather than hacing > > it > > > in the renderer side. > > > When calling MediaCodec, can you check what is the result from > > > format.getInteger(MediaFormat.KEY_CHANNEL_COUNT); > > > if the returned channel count is 1, can you check whether adding a call to > > > MediaFormat.setInteger(MediaFormat.KEY_CHANNEL_MASK, > > > AudioFormat.CHANNEL_OUT_MONO) helps. > > > > KEY_CHANNEL_COUNT is definitely 1. I will try out the setInteger. This is a > > much better solution if it works. > > Setting the channel mask to mono doesn't help. :-( what is the input format? For AAC, you need to pass csd to the MediaCodec, check media_codec_bridge.cc
On 2013/08/02 17:18:40, qinmin wrote: > On 2013/08/02 17:00:51, rtoy wrote: > > On 2013/08/02 16:42:12, rtoy wrote: > > > On 2013/08/02 16:38:08, qinmin wrote: > > > > On 2013/08/02 16:26:00, rtoy wrote: > > > > > On 2013/08/02 16:12:04, qinmin wrote: > > > > > > On 2013/08/02 16:07:22, rtoy wrote: > > > > > > > On 2013/08/02 07:00:53, qinmin wrote: > > > > > > > > I don't think this is appropriate fix. Shouldn't this be fixed in > > > > decoder > > > > > > side > > > > > > > > instead of audio_decoder_android? Fixing it at decoder side helps > us > > > to > > > > > pass > > > > > > > > less data for reading/writing > > > > > > > > > > > > > > Yes, it could be done on the browser (decoder) side. But then the > > > browser > > > > > > will > > > > > > > have to buffer the entire decoded data so that it can make the > > decision. > > > > > > > > > > Currently, the browser just sends the decoded chunks to the > renderer. > > I > > > > > would > > > > > > > prefer not to increase memory usage. > > > > > > > > > > > > > > The current memory usage is: renderer has two copies of the encoded > > > data > > > > > (one > > > > > > > is the original and one for the shared memory buffer) and required > > > > > destination > > > > > > > bus for the decoded data. In addition, the renderer may also have > to > > > > buffer > > > > > > the > > > > > > > decoded data if the size is unknown. > > > > > > > > > > > > > > On the browser, we save a copy of the shared memory buffer into a > file > > > (to > > > > > fix > > > > > > a > > > > > > > different Android issue). There's a small buffer for holding > decoded > > > > > chunks. > > > > > > > > > > > > > > If we change this to do it on the browser side, the browser now gets > a > > > > full > > > > > > copy > > > > > > > of the decoded data. The decoded data is much larger than the > encoded > > > > data, > > > > > > so > > > > > > > we are using much more memory now. > > > > > > > > > > > > But why mediaCodec generates 2 channels instead of 1? I think there > must > > > be > > > > > some > > > > > > configuration problems. > > > > > > And for the decoded chunks, can browser side just remove one channel > > > before > > > > > > returning it to the renderer? > > > > > > > > > > I believe it's a bug in the Android implementation. This doesn't happen > > on > > > > > Nexus devices that I've tested. As for getting the browser to remove > one > > > > > channel, I don't know of any way to tell if the decoded chunk has two > > > channels > > > > > or one channel in it. This is why I check that the final decoded length > > is > > > > > approximately twice the expected length. > > > > > > > > I think we'd like to understand the source of the problem rather than > hacing > > > it > > > > in the renderer side. > > > > When calling MediaCodec, can you check what is the result from > > > > format.getInteger(MediaFormat.KEY_CHANNEL_COUNT); > > > > if the returned channel count is 1, can you check whether adding a call to > > > > MediaFormat.setInteger(MediaFormat.KEY_CHANNEL_MASK, > > > > AudioFormat.CHANNEL_OUT_MONO) helps. > > > > > > KEY_CHANNEL_COUNT is definitely 1. I will try out the setInteger. This is > a > > > much better solution if it works. > > > > Setting the channel mask to mono doesn't help. :-( > > what is the input format? For AAC, you need to pass csd to the MediaCodec, check > media_codec_bridge.cc but since you are using MediaExtractor, this is not needed.
How about after decoding the first buffer, call MediaExtactor.getSampleTime()? And since you have the sample rate and output data size, you can compare the sampletime with output_size/(samplerate * 16bitPCMSize) ?
On 2013/08/02 21:16:53, qinmin wrote: > How about after decoding the first buffer, call MediaExtactor.getSampleTime()? > And since you have the sample rate and output data size, you can compare the > sampletime with output_size/(samplerate * 16bitPCMSize) ? It looks like this will work. The very first buffer is inconsistent, but the getSampleTime on the second and later buffers appear to be consistent and can indicate whether 2 channels or 1 channel has been decoded. I'll need to do some more experimentation and also think about how to handle the case of very short files that only have one buffer. Thanks for this suggestion.
On 2013/08/02 22:15:53, rtoy wrote: > On 2013/08/02 21:16:53, qinmin wrote: > > How about after decoding the first buffer, call MediaExtactor.getSampleTime()? > > And since you have the sample rate and output data size, you can compare the > > sampletime with output_size/(samplerate * 16bitPCMSize) ? > > It looks like this will work. The very first buffer is inconsistent, but the > getSampleTime on the second and later buffers appear to be consistent and can > indicate whether 2 channels or 1 channel has been decoded. I'll need to do some > more experimentation and also think about how to handle the case of very short > files that only have one buffer. > > Thanks for this suggestion. From the latest update on the bug, it turns out that MediaCodec was signaling a change in the output format from 1 channel to 2 and then correctly returning 2 channels. So, it's a bug that webaudio didn't catch this change in output format. However, it's also confusing to the user who specified a 1-channel file to get a 2-channel result. I'm not sure how to solve this now.
PTAL. This new implementation uses the fact that the decoder tells us that the output format has changed. This is used to determine if the second channel should be removed or not.
lgtm % nits https://codereview.chromium.org/21618002/diff/15001/media/base/android/java/s... File media/base/android/java/src/org/chromium/media/WebAudioMediaCodecBridge.java (right): https://codereview.chromium.org/21618002/diff/15001/media/base/android/java/s... media/base/android/java/src/org/chromium/media/WebAudioMediaCodecBridge.java:65: nit: use inputChannelCount https://codereview.chromium.org/21618002/diff/15001/media/base/android/java/s... media/base/android/java/src/org/chromium/media/WebAudioMediaCodecBridge.java:70: nit: outputChannelCount https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... File media/base/android/webaudio_media_codec_bridge.cc (right): https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... media/base/android/webaudio_media_codec_bridge.cc:153: ditto https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... media/base/android/webaudio_media_codec_bridge.cc:154: ditto https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... File media/base/android/webaudio_media_codec_bridge.h (right): https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... media/base/android/webaudio_media_codec_bridge.h:45: input_channel_count https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... media/base/android/webaudio_media_codec_bridge.h:46: output_channel_count
On 2013/08/09 00:31:55, qinmin wrote: > lgtm % nits > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/java/s... > File > media/base/android/java/src/org/chromium/media/WebAudioMediaCodecBridge.java > (right): > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/java/s... > media/base/android/java/src/org/chromium/media/WebAudioMediaCodecBridge.java:65: > > nit: use inputChannelCount > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/java/s... > media/base/android/java/src/org/chromium/media/WebAudioMediaCodecBridge.java:70: > > nit: outputChannelCount > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... > File media/base/android/webaudio_media_codec_bridge.cc (right): > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... > media/base/android/webaudio_media_codec_bridge.cc:153: > ditto > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... > media/base/android/webaudio_media_codec_bridge.cc:154: > ditto > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... > File media/base/android/webaudio_media_codec_bridge.h (right): > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... > media/base/android/webaudio_media_codec_bridge.h:45: > input_channel_count > > https://codereview.chromium.org/21618002/diff/15001/media/base/android/webaud... > media/base/android/webaudio_media_codec_bridge.h:46: > output_channel_count Renamed everything as you suggested. Thanks for the review!
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/rtoy@google.com/21618002/30001
Message was sent while issue was closed.
Change committed as 216779 |