Use postgres to feed LabelStudio

[jalberto]

Is your feature request related to a problem? Please describe.
I will like to use LS to experiment with small datasets (usually with big corpus thou)

Describe the solution you'd like
For a quick start will be great if a SQL query could be defined in LS, then LS will paginate and use each item for tagging

[makseq]

@jalberto Sorry, could you clarify more about your experiment?

SQL query could be defined in LS

What do you mean by this?

[jalberto]

This is my current flow:

• extract data from my main DB into a CSV/TXT
• storage it in some storage/bucket
• connect LS to that bucket
• store results in redis

I will like to be able to do the SQL query directly from LS so data will go directly from my DB to LS and then storage it again in DB

Mostly of ML pipeline systems like kubeflow, expect the data ready, don't seem to take into consideration the labeling part, or at least I cannot find how. so to use SL as a full read-label-store flow for experimentation or small datasets could be useful.

[makseq]

I'm not sure you can do SQL query directly from LS, but you can use two ways to achieve your goal:

1. Django ORM and shell to run something like this.
   Read more here: https://django-extensions.readthedocs.io/en/latest/shell_plus.html
   Try: label-studio shell or label-studio shell < my_script.py
   In your my_script.py you can connect to your DB and populate LS Tasks, e.g.:

ext_data = get_data_from_external_db()
t = Task(data=ext_data, project=project_id)
t.save()

2. Use LS API and similar to (1) my_script.py with API calls.
   Read more: https://api.labelstud.io

[jalberto]

thanks, @makseq I tried 2 but I am not happy with the results (probably my fault), and is it is a bit cumbersome when dealing with biggest chunk of text

I was wondering if, when a new project is created in the backend, it can be configured to read the data from a PG (as it does with redis) that way we can just have a simple SQL query that will run to fetch the data when it requires it

[makseq]

You have to implement your own external storage (Postgres Storage) similar to our storages:
https://github.com/heartexlabs/label-studio/tree/master/label_studio/io_storages

As example your can use redis:
https://github.com/heartexlabs/label-studio/tree/master/label_studio/io_storages/redis

After adding you need to add your app urls here:
https://github.com/heartexlabs/label-studio/blob/master/label_studio/io_storages/urls.py
and add API endpoints here:
https://github.com/heartexlabs/label-studio/blob/master/label_studio/io_storages/all_api.py

PRs are highly appreciated! 🙂

[niklub]

Hi, @jalberto ! That will be an interesting use case we consider while building a python sdk that interacts with Label Studio backend in a less cumbersome way rather than API. Thanks for bringing it up!