Several minutes to transcribe the Beckett sample on Raspberry Pi Zero 2W with tiny model - am I missing anything?

[boutell]

OK, so the Pi Zero 2W has only 512MB of RAM and I realize that's probably too tight for this kind of thing, but I gave it a shot anyway.

Once I figured out you must use the older Bullseye image to install 64-bit Raspberry PiOS Lite on a Pi Zero 2W successfully, I can report it didn't take long to have success with your documented steps.

On my first try the Python process was "Killed," which I pretty much expected (out of memory error). I then added about 2GB of swap space and was able to complete the recognition run.

Alas, it took several minutes at max CPU to transcribe the one-sentence Beckett WAV file.

Results of my run:

(boutell) boutell@sunnyphone:~ $ python
Python 3.9.2 (default, Mar 20 2025, 02:07:39) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import moonshine_onnx
>>> moonshine_onnx.transcribe(moonshine_onnx.ASSETS_DIR / 'beckett.wav', 'moonshine/tiny')
['Ever tried ever failed, no matter try again, fail again, fail better.']

I used the tiny model as you can see.

My guess is this is just the end of the road, I recognize most modern models just aren't built for 512MB of RAM. But I wanted to post my results and see if you have any suggestions before I move on.

Thanks!

[evmaki]

Thanks for testing and sharing your results. This may not be the end of the road! By default, moonshine-onnx.transcribe will load the full-precision fp16 weights. You can try different levels of quantization (and thus smaller models) by initializing the model yourself with the following:

import moonshine_onnx

# load smaller 8-bit quantized model...
model = moonshine_onnx.MoonshineOnnxModel(model_name="moonshine/tiny", model_precision="quantized")

# ...or even smaller 4-bit quantized model
model = moonshine_onnx.MoonshineOnnxModel(model_name="moonshine/tiny", model_precision="quantized_4bit")

moonshine_onnx.transcribe(moonshine_onnx.ASSETS_DIR / 'beckett.wav', model)

Give this a shot and let me know how it goes.

[boutell]

Very interesting! Thanks for engaging.

Here's what I got:

regular tiny: 6.46 x realtime.

"quantized": still requires 3.76 x realtime. Much better, but still too slow.

"quantized_4bit": surprisingly, 4.01 x realtime.

I ran "quantized" a number of additional times, including after letting the pi cool off for a couple minutes. Same result.

On Reddit I also heard from another user who had very good results with a Pi 5 which makes sense. He beat realtime by a long, long way using the regular tiny model. Just acknowledging that this shows it's absolutely viable on a Pi... I'm just trying to use the wimpiest possible modern Pi:

https://www.reddit.com/r/raspberry_pi/comments/1mlq2e8/anyone_using_the_moonshine_voice_recognition/

He reported the process took about 3GB of RAM. I actually don't think it's that high... the highest RSS I was able to detect was around 350MB, even with the full model. However I did get a "Killed" message when I tired to run with no swap available at all.

I tried watching "top" while running "quantized" and for the whole time it was running, I only saw CPU go above 100% once. This makes me wonder if the algorithm is able to utilize multiple cores?