Browser support (discussion)

[mishushakov]

Hey there,
i was able to load the model successfully in browser using tensorflow.js

to convert the models you can use TensorFlow.js converter

pip install tensorflowjs

tensorflowjs_converter --input_format keras models/ts9x/ts9.h5 models/ts9/lite

this issue is a backlog for inferencing and (possibly) training models in browsers

[GuitarML]

Very cool! I'll need to read up more on how that would work, but the fact that it loads is a good start. Also the colab notebook seems to work fairly well for training with the split_data param I added, need to test that more though.

[mishushakov]

Sharing some great progress on the browser runtime:

1. Loaded the models (and weights) into a webpage

2. Modified predict.py to save the X_ordered tensor as json file (for explanation see Prepare data with tensorflow? #5)
   

   GuitarLSTM/predict.py

   Line 58 in 3c1dc9e

   X_ordered = tf.gather(X,indices)

   
   Note: due to big file size i was only able to export one second of the audio (65 megabytes)

3. Loaded the tensor into the TensorFlow.JS and was able to run the prediction
   first one is the input tensor and the second one is the output tensor

   

4. Loaded the resulting float32 data using Audacity's "Import Raw" function

   

The prediction run slowly on my 13-inch MacBook Pro (iGPU), but i will make a test on a PC with GPU
i'd love to share the code once i figure out how to save the bytes as wav, i tried just writing them into a wav file, but didn't work

In the zip file you'll find the in.wav and predict.wav (again, generated through a browser)

Demo.zip

[mishushakov]

actually spent whole night getting that to work, but totally worth it

[GuitarML]

Great stuff, yeah once you get on a roll it's hard to stop, get some sleep!

[mishushakov]

Kept my promise and made the web browser version possible
try out yourself: https://mishushakov.github.com/GuitarLSTM-browser

get the example tensor here (it's so big i had to use git-lfs): https://github.com/mishushakov/GuitarLSTM-browser/raw/master/samples/tensor.json

if you're interested how it all works, feel free to fork the repo: https://github.com/mishushakov/GuitarLSTM-browser

thank you everyone for your help in making this possible
to me this is just the beginning

😃

[mishushakov]

so cool, even works on my iPhone lol

[GuitarML]

@mishushakov Can't wait to try it out! Nice work!

[mishushakov]

on the training side it is possible to convert the the js models to keras (h5)
this means we can train models in the browser and load them into plugin

tensorflowjs_converter --input_format=tfjs_layers_model --output_format=keras models/ts9/model.json models/ts9/ts9.h5

taken from https://github.com/tensorflow/tfjs/tree/master/tfjs-converter#javascript-to-python

[GuitarML]

I think so, but for the plugin I was planning on using json format for the models anyway. H5 is just another data format, as long as all the weights are in the json file we can make it work. And we can work together to make sure the models trained from the browser can be loaded into the future plugin.

H5 is nice because it compresses the data, but I think json is the better option because it's readable in any text editor, and I'm more familiar with loading json in c++.

[mishushakov]

@GuitarML not sure if it's a good idea
give SavedModel a try

once you have that, you can load the model in every version of tensorflow (you read it right, tensorflow, not just keras)
in C++ specifically you can use the tensorflow C++ API or tensorflow Lite C++ API

if this is still isn't enough you could use tensorflow-onnx to convert your model to onnx and load it with onnx runtime

i've been actually wondering whether we can make both networks run in your plugin
one solution would be adding wavenet to the tensorflow version
the other solution would be converting both to onnx