Stuttering mp3 stream on ESP32 — how to optimize or increase buffer? #1930
-
|
I am currently working on a CD player / mini receiver project based around ESP32. As part of that I want to receive internet radio streams, and this library is supposedly perfect for this. However when I try to stream a tiny 128kbps radio in mp3, or a 96kbps AAC, the playback stutters like crazy every 1-2 seconds. When playing a 192kbps mp3 from LittleFS, this does not happen and the sound is perfectly correct. Just in case I also tried turning off all my tasks that are related to UI or HID and it was still stuttering. Similarly with all my UI tasks running the LittleFS playback was all clear. Is there any way I could bump the buffer size to a few megabytes? I have 4 MB of PSRAM and 4 MB of HIMEM if that is also usable. If possible I would prefer to use the whole 4MB of HIMEM as a buffer, but even a 2MB buffer on PSRAM would be massively helpful I think. Here is the snippets of the code. There is no audio related code outside of those except for the router route setup which I included. I2S Output Setup ...
if(next == ROUTE_INTERNAL_CPU) {
spdif_teardown_muting_hax();
_spdif->set_enabled(false);
auto cfg = _i2s->defaultConfig(TX_MODE);
cfg.copyFrom(cpuOutputParams);
cfg.pin_bck = i2s_pins.bck;
cfg.pin_data = i2s_pins.data;
cfg.pin_mck = i2s_pins.mck;
cfg.pin_ws = i2s_pins.lrck;
_i2s->begin(cfg);
_resampler->setAudioInfoOut(cfg);
_resampler->setOutput(*_i2s);
_resampler->begin();
set_mute_internal(false);
}
...The decoder plugs into the converter stream: this is to help if any stream uses a different sample rate from 48000/16: AudioStream * AudioRouter::get_output_port() {
return _resampler;
}When using LittleFS and all sounds perfect: const char *startFilePath="/";
const char* ext="mp3";
AudioSourceLittleFS fileSrc(startFilePath, ext);
...
InternetRadioMode::InternetRadioMode(const PlatformSharedResources res):
urlStream(),
source(urlStream, urls, "audio/mp3"), //<- disregard this for now
codec(),
player(),
sndTask("IRASND", 10000, 5, 1),
station_buttons({}), // todo
chgSource(Button(res.keypad, (1 << 0))),
moreStations(Button(res.keypad, (1 << 7))),
Mode(res) {
rootView = new InternetRadioView({{0, 0}, {160, 32}});
}
void InternetRadioMode::setup() {
AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);
resources.router->activate_route(Platform::AudioRoute::ROUTE_INTERNAL_CPU);
player.setAudioSource(fileSrc);
player.setBufferSize(4096); // adding this had no effect
player.setDecoder(codec);
player.setOutput(*resources.router->get_output_port());
player.addNotifyAudioChange(resources.router->get_output_port());
player.setMetadataCallback(printMetaData);
player.begin();
sndTask.begin([this]() { player.copy(); });
// now sound starts playing and it's perfectly correct
}When using the WiFi stream: const char *urls[] = {
"http://icecast.ndr.de/ndr/ndr1wellenord/kiel/mp3/128/stream.mp3"
};
...
InternetRadioMode::InternetRadioMode(const PlatformSharedResources res):
urlStream(),
source(urlStream, urls, "audio/mp3"), //<- now we use this
codec(),
player(),
sndTask("IRASND", 10000, 5, 1),
station_buttons({}), // todo
chgSource(Button(res.keypad, (1 << 0))),
moreStations(Button(res.keypad, (1 << 7))),
Mode(res) {
rootView = new InternetRadioView({{0, 0}, {160, 32}});
}
void InternetRadioMode::setup() {
AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);
resources.router->activate_route(Platform::AudioRoute::ROUTE_INTERNAL_CPU);
player.setAudioSource(source);
player.setBufferSize(4096); // adding this had no effect
urlStream.setReadBufferSize(2 * 1024 * 1024); // adding this had no effect either
urlStream.setPowerSave(false); // this is the default but just in case -- still, no effect
player.setDecoder(codec);
player.setOutput(*resources.router->get_output_port());
player.addNotifyAudioChange(resources.router->get_output_port());
player.setMetadataCallback(printMetaData);
player.begin();
sndTask.begin([this]() { player.copy(); });
}I presume the buffering is the issue since I can listen to the same stream perfectly on my PC. Thanks in advance for your help P.S. For some reason using audio also kills JTAG... but not sure if it's an ESP thing or this library's issue. |
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 24 replies
-
|
Just in case I did a speed test against the same stream: void InternetRadioMode::setup() {
AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);
MeasuringStream measure(50, &Serial);
StreamCopy copier(measure, urlStream);
urlStream.begin("http://icecast.ndr.de/ndr/ndr1wellenord/kiel/mp3/128/stream.mp3");
while(1) copier.copy();
...
}Result: Looking at the numbers in the very first section of Phil's post — seems like it's average for a radio station? (I mean, 16000 bytes per second is literally 128kbps which is what this station broadcasts at) So probably the issue is not in the connection speed? |
Beta Was this translation helpful? Give feedback.
-
|
The problem is partially remedied by an enormously oversized buffer at the output side, as well as disabling the resampling — the stutter becomes fairly rare (once every multiple seconds rather than once every 1-2 seconds): But eventually something in the codec shits itself and the system crashes, probably for the lack of RAM: |
Beta Was this translation helpful? Give feedback.
-
|
can you show more of the error? I am a noobie but when I have found memory issues and I try to do something like this after a lot of memory intensive tasks like b4 the stream starts etc., again noob advice but maybe can give it a swing while you wait on Phil to give out the real advice lol but I don't think you can hurt adding this anyways, the delay helps a bit too, as so much get bottlenecked with so much going on. I'm sure phil has similar in his library in crucial spots, so it could be over kill, not sure |
Beta Was this translation helpful? Give feedback.
-
|
That is the whole error output, otherwise I do have a similar thing albeit without a delay. The delay only helps in a way to give the CPU time to bum the watchdog, ideally it should not be needed, if without it your firmware crashes one of the tasks might be hogging too much of the time — better delete it if not needed or lower its priority to let the idle tasks do their job too. |
Beta Was this translation helpful? Give feedback.
-
|
well sorry i was of not any help, prob got all excited you got a reply too :( |
Beta Was this translation helpful? Give feedback.
-
|
No worries! |
Beta Was this translation helpful? Give feedback.
-
|
I would suggest to check first with this simple sketch, if you can reproduce the issue with your url. If this is playing fine, then the issue must be somewhere in your logic and e.g. your playback task is slowed down too much. To address the issue with the metadata, I suggest that you filter it out e.g. with a MetaDataFilterDecoder or a MetaDataFilter in your chain. It would be interesting if you could debug this error, so that we can try to prevent this in the future: The findTag method currently has already some checks to prevent this and I am not sure what would be missing... You can also try to use one of the Buffered URL Streams (which buffers the encoded data) that I initially implemented, but never needed because the regular URLStream was always good enough... |
Beta Was this translation helpful? Give feedback.
-
|
The setReadBufferSize is only relevant if you read individual bytes by calling read() instead of readBytes() |
Beta Was this translation helpful? Give feedback.
-
|
Make sure that you have PSRAM activated, otherwise the data is allocated in RAM. In RAM an allocation one big buffer might fail, but many small buffers will work.... |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I have PSRAM enabled and use it extensively for my own things such as display fonts decompression. However it seems most of the memory used is indeed the RAM: Nothing too catastrophic since I can put UI stuff in PSRAM, but not sure if this is bad. P.S. one more nitpick: |
Beta Was this translation helpful? Give feedback.
-
|
I guess that's WIFI which uses a lot of RAM and not my buffers that should really use PSRAM when activated. Not sure if this is working but did you try this to move more standard functionality to PSRAM ? Lambdas w/o capture should work... |
Beta Was this translation helpful? Give feedback.
-
|
Without capture yes, but I am in a context of an instance so I need a capture and those obviously don't 😛 (I will be deallocating the instance when the user switches from Radio to Bluetooth or CD-Audio to free up memory, so I cannot make the object static) I resorted to including a static void with a pointer for the time being so not a blocker. Inre the PSRAM limit setting, I actually put that in always at 128 bytes threshold, but during the debugging today I turned it off just in case PSRAM is slow enough to cause this stuttering. I'll try putting it back now. Upd: yes, it freed up some more memory, it's now on par with the same numbers in the CD-Audio mode. Stuttering is not happening for the most part too: And sorry, but yet another batch of questions to wrap this thread up:
I think I've also found a minor bug in the libmad implementation, put it here: pschatzmann/arduino-libmad#2 |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Inre 2, so all the audioplayer does is playlist handling logic for things like back/forward, nothing related to signal processing? For 3, I'll take a look at it. Does that mean starting the copier first and only then connecting an appropriate output to it? For 4, I think maybe if when a stream error occurs the pipeline gets stopped I could check isActive, and restart the pipeline if needed. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I added some DRAFT implementation for a MultiDecoder that should be able to support multiple decoders |
Beta Was this translation helpful? Give feedback.
-
Sounds like this works good enough for me because I will need to restart the pipeline with a new URL anyway. But I don't exactly understand how would it choose the decoder, do I just connect them like so?
This looks good too, do I just subclass it to add a list of my decoders and that's it? |
Beta Was this translation helpful? Give feedback.
-
|
You would just set/change the decoder: encoded_audio_stream.setDecoder(decoder); and call encoded_audio_stream.begin(). The MultiDecoder has an addDecoder() method where you add the decoder and it's mime type |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, I'm getting a bunch of build errors now for the multidecoder, am I missing a build flag or something? Also the addDecoder method is private |
Beta Was this translation helpful? Give feedback.
-
|
Try to update again: I tested with the following sketch #include "AudioTools.h"
#include "AudioTools/AudioCodecs/CodecMP3Helix.h"
#include "AudioTools/AudioCodecs/CodecAACHelix.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
URLStream url("ssid","password");
AudioBoardStream i2s(AudioKitEs8388V1);
MultiDecoder multi;
MP3DecoderHelix mp3;
AACDecoderHelix aac;
WAVDecoder wav;
EncodedAudioStream dec(&i2s, &multi); // Decoding stream
StreamCopy copier(dec, url); // copy url to decoder
void setup(){
Serial.begin(115200);
AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);
multi.addDecoder(mp3, "audio/mpeg");
multi.addDecoder(aac, "audio/aac");
multi.addDecoder(wav, "audio/vnd.wave");
// setup i2s
auto config = i2s.defaultConfig(TX_MODE);
// you could define e.g your pins and change other settings
//config.pin_ws = 10;
//config.pin_bck = 11;
//config.pin_data = 12;
//config.mode = I2S_STD_FORMAT;
i2s.begin(config);
// setup I2S based on sampling rate provided by decoder
dec.begin();
// mp3 radio
url.begin("http://stream.srg-ssr.ch/m/rsj/mp3_128","audio/mpeg");
}
void loop(){
copier.copy();
} |
Beta Was this translation helpful? Give feedback.
-
|
The build error is still there unless I remove the default values from DecoderInfo. Am I missing a C++ version? |
Beta Was this translation helpful? Give feedback.
-
|
What esp32 core version are you using ? The actual is 3.1.3. |
Beta Was this translation helpful? Give feedback.
-
|
OK, I tested and a few issues popped up. The setup is like this: StreamCopy copier;
MultiDecoder codec;
MP3DecoderHelix mp3;
AACDecoderHelix aac;
ICYStreamBuffered urlStream;
EncodedAudioStream decoder;void InternetRadioMode::play() {
if(playTask != nullptr) {
playTask->end();
delete playTask;
playTask = nullptr;
}
rootView->set_loading(true);
playTask = new Task("IRASND", 12000, 15, 1);
playTask->begin([this]() {
decoder.setOutput(resources.router->get_output_port());
decoder.begin();
urlStream.begin("http://srv01.gpmradio.ru/stream/reg/mp3/128/region_relax_86");
codec.addNotifyAudioChange(*resources.router->get_output_port());
// codec.setMetadataCallback(_update_meta_global);
copier.begin();
rootView->set_loading(false);
while(1) {
copier.copy();
}
});
}The issues are as follows:
Upd: yes, it seems it's related to the AAC codec in Helix, as I am not using multi decoder now and while MP3 works fine, AAC sounds sped up and crashes the ESP32 all the time. Increasing stack to 80000 did not fix this as well Upd2: the same crash sometimes happens with a Helix MP3 decoder too. E.g. when using |
Beta Was this translation helpful? Give feedback.
-
|
I could not reproduce your issue with the notifications in my tests with the latest code: 16:28:01.853 -> [I] I2SConfigESP32V1.h : 102 - pin_ws: 25 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it seems to be fine now with the MP3 streams and some AAC stations, but some other AAC stations sound weirdly high pitched. Example:
|
Beta Was this translation helpful? Give feedback.
-
|
Hmm, so for the time being I decided to work on refactoring the code to not use AudioPlayer and instead rely on just a StreamCopy. Surprisingly, the stutter is back — but this time it's local. I tried enabling the logs for the thing, and what I noticed is: Every time the MetaDataICY string is logged, the audio stutters, all other times it's fine. Could it be that the Player is using a different way to parse metadata which isn't causing such lag? If I change it to a normal URLStreamBuffered, it doesn't lag anymore, but of course doesn't give metadata too. But I still can't grasp why it worked via AudioPlayer and with a simple StreamCopy it broke. UPD: I fixed it by specifying a buffer size for the ICY stream: |
Beta Was this translation helpful? Give feedback.
-
|
On the latest I'm still getting occasional stack corruption with some mp3 and most AAC stations. Here is how I play now to keep the UI thread unblocked: sndTask = new Task("IRASND", 30000, 15, 1);
sndTask->begin([this]() {
decoder.addNotifyAudioChange(*resources.router->get_output_port());
urlStream.setMetadataCallback(_update_meta_global);
copier.resize(24 * 1024);
decoder.begin();
urlStream.begin("http://icecast.ndr.de/ndr/ndr1wellenord/kiel/mp3/128/stream.mp3", "audio/mpeg");
rootView->set_loading(false);
int hwmark = 9999999;
while(1) {
copier.copy();
int x = uxTaskGetStackHighWaterMark(NULL);
if(x < hwmark) {
hwmark = x;
ESP_LOGW(LOG_TAG, "High water mark of audio task is now %i", hwmark);
}
}
});What happens, sometimes after a minute, sometimes near instantly: |
Beta Was this translation helpful? Give feedback.
-
|
If I use regular ICYStream instead of the buffered one, this doesn't happen, but then the stream stutters like crazy |
Beta Was this translation helpful? Give feedback.
-
|
Finally seems to be relatively stable with the following configuration: Used components: StreamCopy copier;
MP3DecoderHelix mp3;
AACDecoderHelix aac;
MultiDecoder multicodec;
EncodedAudioOutput decoder; //<- What is the difference from EncodedAudioStream? The Stream is stuttering, the Output is not
ICYStream urlStream;
Task * sndTask;Initialized as follows: Playback as follows: void InternetRadioMode::play() {
rootView->set_loading(true);
if(sndTask != nullptr) {
delete sndTask;
sndTask = nullptr;
}
sndTask = new Task("IRASND", 4096, 17, 1);
sndTask->begin([this]() {
decoder.addNotifyAudioChange(*resources.router->get_output_port());
urlStream.setMetadataCallback(_update_meta_global);
decoder.begin();
urlStream.begin("http://icecast.ndr.de/ndr/ndr1wellenord/kiel/mp3/128/stream.mp3", "audio/mpeg");
rootView->set_loading(false);
while(1) {
copier.copy();
taskYIELD();
}
});
}Seems to only have minor stutters (probably network underruns), and not starving the TCP/IP stack of CPU time. Key differences:
I guess at this library complexity things like this become more like black magic, so I hope this thread helps someone else later down the line, even though it's all over the place. About the stream that sounds sped up: The other AAC stream If I bypass MultiDecoder and go directly into AACDecoderHelix, it plays, albeit stutters. Sometimes it fails entirely with All in all, some logging that could show where buffer underruns happen (network/input, decoding, output) would be very helpful. One more concern is that the URLStream does not use a circular buffer but instead waits for the whole buffer to be consumed before reading more from HTTP. This causes a potential issue where the decoder ran out of data but the URLStream read buffer has only a handful of bytes remaining, so the decoder now has to wait until the HTTP read continues. |
Beta Was this translation helpful? Give feedback.
I would suggest to check first with this simple sketch, if you can reproduce the issue with your url.
If this is playing fine, then the issue must be somewhere in your logic and e.g. your playback task is slowed down too much.
To address the issue with the metadata, I suggest that you filter it out e.g. with a MetaDataFilterDecoder or a MetaDataFilter in your chain. It would be interesting if you could debug this error, so that we can try to prevent this in the future: The findTag method currently has already some checks to prevent this and I am not sure what would be missing...
You can also try to use one of the Buffered URL Streams (which buffers the encoded data) that I initially imple…