Some Ideas to get GuitarML into 2022

[pgarz]

Firstly, let me introduce myself. I'm Pedro Garzon, professionally an ML engineer at a startup doing AI for scientific simulations. I have a background in AI from Stanford as well. More importantly, I've had the same idea of making a neural guitar pedal before. So glad someone started the groundwork foundations! I've spent most of my free time over quarantine just practicing guitar and exploring tones. I've been catching up on DL for audio and I really think 2022 can be a break-out year to making a supremely affordable neural pedal. I've got some ideas

Here are some technical infrastructure ideas

Build PyTorch support

• PyTorch has the capability of re-writing models trained with Python into compilable C++. This can be a huge boost for not having to manually re-write all models in raw C++ or having to rely on RTNeural. With going from Pytorch -> TorchScript -> C++, the C++ to run a loaded model might be as simple as this

std::vector<torch::jit::IValue> inputs; inputs.push_back(torch::ones({1, 3, 224, 224})); // Execute the model and turn its output into a tensor. at::Tensor output = module.forward(inputs).toTensor();

• One could also convert PyTorch to ONNX and run the models on the ONNX C++ runtime, which can be faster at smaller batch sizes

• It appears one could load PyTorch directly on a Rasberry Pi, but the hardware costs go up considerably. Instead it might make more sense to build for on computer processing https://qengineering.eu/install-pytorch-on-raspberry-pi-4.html

• Build-in support for M1 chip training

• It seems the M1 series of chips have a bright future in audio production: https://www.waves.com/apple-m1-for-music-producers

• It's already possible to run Pytorch directly on M1 chips https://towardsdatascience.com/yes-you-can-run-pytorch-natively-on-m1-macbooks-and-heres-how-35d2eaa07a83

• Leveraging M1 computation could be a computational breakthrough and squeezing real-time performance out of more complicated network architectures.

• With PyTorch, we could take advantage of Pytorch Lightning as well and highly organize the data-loading and model training code to be extremely modular and readable

• Support for on local computer recording

• We're quite limited by the current memory/CPU capabilities of the RasberryPi setup. So it would be ideal to also have Pytorch models be capable of running locally on the host computer running a DAW for recording

• There also might be a way to use the RasberryPi as the main interface for the guitar. So we could have a chain going guitar -> RasberryPi -> host computer for processing -> back to RasberryPi -> real-world amp

Logic Pro X support

• I personally use Logic, so would love to use my DAW of preference. Today I tried it out on Logic with the NueralPi plugin and it seemed to work just fine. Hopefully there aren't any more quirks to figure out there, but I have a stake in keeping the plugin interface working in Logic

And here are some very brief avenues of exploring richer network architectures:
Transformers, VQ-VAE, WaveNet-like architectures, Diffusion models, Fourier Transform tricks

All of which have PyTorch implementations!

Eager to hear feedback! Def the first step is point number building out a tighter PyTorch infrastructure

[GuitarML]

@pgarz Thanks for all the ideas! Using the Pytorch models directly might make it more flexible on the ML side, but I like using RTNeural because it was specifically made for audio, and the benchmarks show that for smaller models it is significantly faster for real-time. I'm definitely not opposed to testing things out, or even making a full plugin to show how it can be done. Using the M1 chipset to optimize training would be awesome. Basically I'm for any of those things you would like to do, but I would want to fully test things out and ensure it offers a performance or feature improvement for the end user before implementing into the official GuitarML main branches. It might be beneficial to create a new repo to test using a fully Pytorch workflow, or when trying new model architectures.

The current repo I use for ML training is: https://github.com/GuitarML/Automated-GuitarAmpModelling

Not sure if you were looking at PedalNetRT, but I haven't touched that in awhile. The above repo contains support for wavenet training and I'd recommend starting there, but I do like the pytorch-lightning implementations when feasible.

The more DAW's we can support the better, Logic is one I definitely want to keep supporting. I didn't have any issues running in Logic when I had the free trial, but if you find anything you can create a new issue.

Thanks and let me know what you think!

[pgarz]

@GuitarML Thanks for your thoughts!

Your word lead me to do some more digging that I missed on my first round of scoping this out. Mainly, I totally missed how the Pytorch -> C++ converter already supported by the Pytorch repo is below real-time performance for 1D convolutions, a huge misfortune for my Pytorch-based approach. However, it seems RTNeural really has the right idea and has already done some of the work that will inevitably happen to obtain real-time performance.

My detective work leads me to believe in the future Pytorch will implement some fast Pytorch -> Apple Accelerator / BNNS C++ converter. They are actively hiring for this position: https://jobs.apple.com/en-us/details/200265506/accelerating-pytorch-on-macs-with-bnns

This might take a while, but at least BNNS offers some more abstraction that could simply be the Apple Accelerate flavor of RTNeural that is already in place. Albeit, of the cost of being Apple ecosystem contained, it could be the easiest option for now for avoiding re-writing some of the lower-level layers, and focusing on experimenting with more conceptually intriguing architectures.

I might take this as my path of exploration for now in attempts of trying to squeeze the use of some real-time capable Transformer
/ fast attention mechanism architecture.

No doubt the results with distortion is working well, but the longer time-based effects will certainly be quite the challenge.

[GuitarML]

@pgarz Indeed! Time-based real-time with neural nets will be a challenge. Any small optimizations in processing the neural nets would be a huge benefit, since we're interested in running the net at 44.1kHz or higher. Also, knowing that we're touching on the same subjects as Apple's research is very exciting.