Bufferscaler implementation with the Libsamplerate callback API #15005

6charm · 2025-06-25T11:39:12Z

Samplerate "linear" and "fastest sinc" now both work for scratching without artefacts. "Best quality" has underflows during scratch, but may be usable for moderate sample-rate conversions such as recording/broadcast wrt issue #10611

…oose scratching and keylock libraries.

…ling. TODO: GUI scratching incomplete, formal audio quality testing

…layback, tempo ramping.

…ase_rate conversion, scratching. Comments welcome.

daschuer · 2025-06-25T14:37:35Z

res/skins/LateNight/library.xml

@@ -31,14 +31,11 @@
                <SizePolicy>min,me</SizePolicy>
                <MinimumSize>100,</MinimumSize>
                <Children>
-


Can you revert this change. You may use a "--fixup" commit.

daschuer · 2025-06-25T14:42:57Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+    case EngineBuffer::ScratchingEngine::SampleRateSincFinest:
+        src_quality = SRC_SINC_BEST_QUALITY;
+        break;
+    }


DEBUG_ASSERT will abbort Mixxx during debug and will do nothing in release builds:

Suggested change

}

case EngineBuffer::ScratchingEngine::NaiveLinear:

case default:

DEBUG_ASSERT(!"unsupported quality setting");

}

daschuer · 2025-06-25T14:51:19Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+
+    if (error || !m_pResampler) {
+        qWarning() << "libsamplerate initialization error:" << src_strerror(error);
+        m_pResampler = nullptr;


Suggested change

m_pResampler = nullptr;

if (m_pResampler) {

free(m_pResampler);

m_pResampler = nullptr;

}

daschuer · 2025-06-25T14:52:28Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+             << "Effective rate:" << m_effectiveRate;
+}
+
+// Callback API requires two callbacks: THis one is


Suggested change

// Callback API requires two callbacks: THis one is

// Callback API requires two callbacks: This one is

daschuer · 2025-06-25T14:54:35Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+// data: SRC_CB_DATA
+// ---
+// Needs to be a static function, since it is called by libsamplerate core.
+static long raman_get_input_frames_cb(void* cb_data, float** audio) {


Please move it to the anonymous namespace above. This is the same than the static keyword.

daschuer · 2025-06-25T14:59:32Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+    return pThis->getInputFrames(audio); // public
+}
+
+long EngineBufferScaleSR::getInputFrames(float** audio) {


This is a output parameter and requires a pointer to an array of float samples. "ppAudio".
libsamplerate does not maintain its own buffer.

daschuer · 2025-06-25T15:02:53Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+
+long EngineBufferScaleSR::getInputFrames(float** audio) {
+    SINT required_input_frames = static_cast<SINT>(
+            std::ceil(m_outFrames / m_srcRatio)); // can be larger than backbuffersize.


getInputFrames call is repeated until all required frames are available. This is likely more than this calculation because sinc needs some settling samples. So we probably can instead pass a whole chunk without much worrying.

daschuer · 2025-06-25T15:13:28Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+            static_cast<SINT>(m_bufferBack.size()));
+
+    SINT samples_read = m_pReadAheadManager->getNextSamples(
+            m_effectiveRate,


Do we inform the RAMAN that we may want backward?

mixxx/src/engine/bufferscalers/enginebufferscalerubberband.cpp

Line 325 in 55dcd1e

// are going forward or backward.

RAMAN getnextsamples() method uses the sign of m_effectiverate to infer direction.
m_effectiverate is always signed here, so an explicit check is not necessary.

The rubberband line linked is simply calculating effective_rate

daschuer · 2025-06-25T15:42:49Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+        *audio = nullptr;
+        return 0;
+    }
+    m_inputFramesRead = getOutputSignal().samples2frames(samples_read);


m_inputFramesRead returned by scaleBuffer() shall only return the actual read samples in the engine, not the buffered samples.

Oh yes, my mistake.
The problem is that libsamplerate callback api does not explicitly return #input samples consumed to caller.

We need to track how many frames we have passed via the callback and how many
state->saved_frames we had at the beginning of scaleBuffer and after. That's straight forward and works probably reasonable good.

Will implement this ^ to get an accurate m_inputFramesRead count.

daschuer · 2025-06-25T16:22:06Z

src/engine/bufferscalers/enginebufferscalesr.cpp

+    const int framesProduced = do_scale(pOutputBuffer, m_outFrames);
+    assert(framesProduced == m_outFrames);
+
+    return m_inputFramesRead;


This here starts to get complicated. We need to track how many frames we have passed via the callback and how many
state->saved_frames we had at the beginning of scaleBuffer and after. That's straight forward and works probably reasonable good.

But to make it work perfectly we need double precision. We need create the fractional part, because the user may stop at an output sample that is just in between two input sample. (If you consider playing with half speed)
The libsamplerate interface cant do this ... Unfortunately ... so we may consider to fork libsamperate for adding the double precision interface.

Maybe this rounding down to long nature is the cause for the remaining clicks? .. Unsure ...

But to make it work perfectly we need double precision. We need create the fractional part, because the user may stop at an output sample that is just in between two input sample. (If you consider playing with half speed)
The libsamplerate interface cant do this ... Unfortunately ... so we may consider to fork libsamperate for adding the double precision interface.
Maybe this rounding down to long nature is the cause for the remaining clicks? .. Unsure ..

I will correct the m_inputFramesRead value first and observe. Just to confirm, are the clicks you hear reproduceable? Or random? On my machine, most times playback occurs without audible clicks.

Maybe this rounding down to long nature is the cause for the remaining clicks? .. Unsure ..

So we'd have to submit a PR to libsamplerate with our proposed double interface? Or is it more common to distribute as a submodule?

I see tree options:

run a second model that keeps track of the position

what you do here right now.

copy the relevant code from libsamplerate to our lib folder and make the required variables double

We may later contribute it upstream

rip off the sinc algorithm of libsamplerate and use it here. Make it our own.

More easy to maintain, but we are cut from upstream fixes

Access private interface of libsamplerate

This works on Linux, unsure if it works on Windows.

The private interface is here:
https://github.com/libsndfile/libsamplerate/blob/15c392d47e71b9395a759544b3818a1235fe1a1d/src/common.h#L155

Wow :-) "last_position" is the double precision value that we need.
Let's copy the "struct SRC_STATE_tag" from common.h to our code and file a small wrapper src_get_last_position().
But first we need to check src_get_version() and we may also do a sanity check in addition and verify for example the address in "user_callback_data".

Okay, will get this done asap

Okay how about this:

I implement @daschuer "hack" immediately to test audio quality during resample. I still need to write scratching unit tests.

Submit a PR to libsamplerate with a get_last_frame() and double-precision interface.

Do not merge this pr into mixxx upstream until above libsamplerate changes are merged to libsamplerate upstream.

While waiting for libsamplerate to be merged, I can implement bufferscalers using different resample libraries that may be merged to mixxx upstream.

@Swiftb0y @daschuer Does this seem like a good plan? Also if possible, could you please test this latest commit for audio quality issues? The automated enginebufferE2E tests fail, but values are very close to the "golden samples".

right, so your solution is already dead on arrival...

You probably misread the comment. The braking comit has been introduced with the mayor version jump from 1.9 to 2.0. this was 5 years ago. Since then it is stable up to HEAD.

There is no other ready to use solution in sight, that is provided a as a general purpose library. Other solution likely require to distribute it outsrlf anyway.

Now we need here a POC that sinc works for scratching first. So we are in a good route.

But what do you think about the static link idea to pin a known working version?

So we need to make it "defined". This can be done by statically link to a specific version.

No, locking us into a specific version will be a maintenance nightmare for us and our packager maintainers. This is not a reasonable solution.

Again, do what you gotta do for a PoC, but don't introduce that stuff into main. The only proper solution I see is to make that API stable upstream and discuss with the upstream devs/maintainers if that is reasonable for them.

@Swiftb0y Please use "Request Changes" is such cases, otherwise such code can slip into the tree

Wow :-) "last_position" is the double precision value that we need.
Let's copy the "struct SRC_STATE_tag" from common.h to our code and file a small wrapper src_get_last_position().
But first we need to check src_get_version() and we may also do a sanity check in addition and verify for example the address in "user_callback_data".

The actual attribute is state->priv->in_used. state->last_position always returns a double < 1.
Example: src_linear.c:52::linear_vari_process(): data->input_frames_used = priv->in_used / state->channels ;
(similar for sinc, with some modifications.)

Just to confirm, we should returnstate->priv->in_used + state->last_position?

…Comments on audio quality would be helpful.

Swiftb0y

See #15005 (comment)

daschuer · 2025-06-26T23:33:55Z

It is getting better. Playback at different ratios works nice. Also rate transitions are smooth at any algorithms.
I might hear a small click now and then when changing ratio with all and when crossing zero with algorithms.

I have added this hack:

#include "common.h" 
....
qDebug() << "last position" << m_pResampler->last_position;

And get this output.
So it looks like last_position behaves different than expected.

debug [0x58a6ad9febf0] Wrote  128  to DAC
debug [0x58a6ad9febf0] last position 0.04
debug [0x58a6ad9febf0] src ratio:  0.925926
debug [0x58a6ad9febf0] Need  138.24  frames from RAMAN.
debug [0x58a6ad9febf0] Wrote  128  to DAC
debug [0x58a6ad9febf0] last position 0.28
debug [0x58a6ad9febf0] src ratio:  0.925926
debug [0x58a6ad9febf0] Need  138.24  frames from RAMAN.
debug [0x58a6ad9febf0] Wrote  128  to DAC
debug [0x58a6ad9febf0] last position 0.52
debug [0x58a6ad9febf0] src ratio:  0.925926
debug [0x58a6ad9febf0] Need  138.24  frames from RAMAN.
debug [0x58a6ad9febf0] Wrote  128  to DAC
debug [0x58a6ad9febf0] last position 0.76

The approach with the upstream patch is a good idea. Hopefully we get it merged soon, to not block our project on that.

daschuer · 2025-06-27T00:36:07Z

the linear resampler keeps a "last_value"

It points to the first input frame after reset. last_position is zero. This means the first frame is duplicated which is kind of not correct. But we need one frame extra.
It becomes the last frame of input_frames_used before returning.
last_position is the fractional part of frames used ahead of last_position.

So we have in the pass through case a last_position of 0 and the input_frames_used is the truth.
Wen we have called before last_position can have any value but is not changed during the call.
Let's assume input_frames_used = 4 and last_position = 0,3

L--|--|--|--|  `input_frames_used` L = `last_value` 
 |--|--|--|  `output

This means we are up to 1 frame delayed.
The actual consumed samples in the following runs are
1 - last_position before + input_frames_used - 1 + last_position after
simplified to -> input_frames_used + last_position after - last_position before

Unfortunately with the callback interface input_frames_used is not accessible. But I think you can recreate it from the passed buffers.

6charm · 2025-06-27T19:03:34Z

src/engine/bufferscalers/enginebufferscalesr.cpp

m_inputFramesRead = 0.0 in scaleBuffer()

The comment is misleading, it is from a older version where i set m_inputFframesRead = m_pResampler->saved_state.

daschuer · 2025-06-27T22:29:14Z

Well done. The Linear rescaler works now with libsamplerate.

Sinc has still issues. It is good notable if you want to hold the waveforms with the mouse. Than the sound is cycling around the hold position. I think we hear the settling samples, the sinc algorithms requires.

I have confirmed by recording the input buffer of a 4410 Hz rectangular signal, that the output of libsamplerate is one frame delayed. I have recorded the original signal left and the scaled signal right and then watched the samples in audacity.

We may correct that by doing a Dummy call consuming this one sample.

Fastest sinc is exactly 512 frames behind. Instead of silently duplicating one sample, it requests this 512 buffer explicit during the first call. This way the it the samples are played at the right moment, but our transport calculation is tricked. I think we need report 512 consumed frames less during the first call after reset.

I did a brief measurement of resampling a 48 kHz Track to 44.1 and a 5,8 ms buffer and got this is result:

448 µs Best
57 µs Fast
10 µs linear
20 µs internal

This means we may either completely fix the linear version of libsamplerate and retire our internal linear resampler or apply the same trick libsamplerate used to speed it up the internal.

6charm · 2025-06-30T19:07:50Z

Thank you!

We may correct that by doing a Dummy call consuming this one sample.

This is done, will also push a commit with GUI changes for record+broadcast.

Sinc has still issues. It is good notable if you want to hold the waveforms with the mouse. Than the sound is cycling around the hold position. I think we hear the settling samples, the sinc algorithms requires.

Currently working on the fix. Will have this ready with the final recording interface.

armaan added 8 commits May 28, 2025 00:35

WIP

719fb7b

explicit linking libsamplerate

2adebd4

wip: add libsamplerate scaler. TODO: scaling logic

0060b44

Added a GUI box in preferences that enables users to independently ch…

1f3fbd3

…oose scratching and keylock libraries.

First implementation of a libsamplerate-based scaler class for resamp…

5cf32dc

…ling. TODO: GUI scratching incomplete, formal audio quality testing

Minimal libsamplerate scaler: backwards scratching support, reverse p…

d1425ad

…layback, tempo ramping.

Sinc Linear fixed, need comments on audio quality

f77e58e

Libsamplerate bufferscaler with callback API. Tested tempo ramping, b…

6a41542

…ase_rate conversion, scratching. Comments welcome.

github-actions bot added skins engine ui build preferences packaging soundio cmake developer experience Issues, bugs and PRs related to the development process, development environment & developer docs labels Jun 25, 2025

Merge branch 'main' into temp-samplerate-callback-api

cc96cbf

daschuer requested changes Jun 25, 2025

View reviewed changes

armaan added 2 commits June 26, 2025 16:38

minor fixup

7c2d089

libsamplerate bufferscaler implemented with reduced backbuffer size. …

d28df14

…Comments on audio quality would be helpful.

Swiftb0y requested changes Jun 26, 2025

View reviewed changes

libsamplerate hack with fractional input frames read

ede79a2

6charm commented Jun 27, 2025

View reviewed changes

6charm mentioned this pull request Jul 15, 2025

Double precision interface for the caller to calculate frames consumed while using the callback API libsndfile/libsamplerate#226

Open

-    }
+   case EngineBuffer::ScratchingEngine::NaiveLinear:
+   case default:
+       DEBUG_ASSERT(!"unsupported quality setting");
+    }

	// Callback API requires two callbacks: THis one is
	// Callback API requires two callbacks: This one is

Uh oh!

Bufferscaler implementation with the Libsamplerate callback API #15005

Are you sure you want to change the base?

Bufferscaler implementation with the Libsamplerate callback API #15005

Uh oh!

Conversation

6charm commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

6charm Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

6charm Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

6charm Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

6charm Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

6charm Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiftb0y left a comment

Choose a reason for hiding this comment

Uh oh!

daschuer commented Jun 26, 2025

Uh oh!

daschuer commented Jun 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daschuer commented Jun 27, 2025

Uh oh!

6charm commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

6charm commented Jun 25, 2025 •

edited

Loading

6charm Jun 25, 2025 •

edited

Loading

6charm Jun 25, 2025 •

edited

Loading

6charm Jun 25, 2025 •

edited

Loading

6charm Jun 26, 2025 •

edited

Loading

6charm Jun 26, 2025 •

edited

Loading

6charm commented Jun 30, 2025 •

edited

Loading