How to create a new extension

[cperezmig]

Hi, I am thinking on a possible use case to DuckDB that I would like to test: to use mlpack (or any other ML framework) to add ML functions to DuckDB, being able to train and use models over data with an SQL request. I think this could be implemented with UDFs or as a new extension, but I can't find any documentation about how to develop a new extension. Is the documentation going to cover this part in the future? Could anyone give me a hint about how to start?

Thanks in advance

[Mytherin]

There's a simple example extension in the test/extension directory. We are in the process of adding support for installing external extensions and moving several of our own extensions to external extensions, so expect more work/documentation done there in the future.

[fire]

Hi, I am also interested in doing this.

My idea is to integrate https://github.com/fire/qmf (my fork of Quora's bpr library for ranking choices).

The idea is to support calculate bpr over [uuid, uuid, weight] and [numeric, numeric, weight] as a duckdb extension.

[cperezmig]

There's a simple example extension in the test/extension directory. We are in the process of adding support for installing external extensions and moving several of our own extensions to external extensions, so expect more work/documentation done there in the future.

I see from the example you give that it consists in registering a new UDF which I think serves for my purpose. Is it possible from this UDF to modify the DB performing an insert/update?

[Mytherin]

You could insert and update rows to a table inside a UDF, yes, although a scalar UDF is not intended for this. Perhaps a CALL function similar to the TPC-H extension is more suitable for this, then you can do e.g. CALL my_function(...).

[cperezmig]

Some questions about "CALL". Where can I find documentation about it? What are the possible parameters to a CALL method? Only scalar values? It is possible to call a vectorized method with CALL and pass to it the results from a SELECT statement?

My idea: to use CALL to train a ML model from a table or a SELECT statement. Once trained, store the model serialized in a custom table, thus my question about making an INSERT, and use a vectorized UDF to make predictions over tables using the deserialized model. I really don't like the idea of storing the model in a table, but it seems to me as the only possible solution to keep models inside the DB.