Refactor as a (self hosted) service

[rom1504]

https://github.com/rom1504/img2dataset/tree/streaming_refacto some work I started on that some 8 months ago
I still think it's the right direction

may try to finish it soon

would close #82 #188 and #135

[rom1504]

in term of implementation, maybe ray eg #272 can help guide things / compare with the http path

[rom1504]

https://github.com/ml6team/fondant is doing some good things in term of packaging, a bit similar to what jina is doing. They're doing docker though which is not clear how it can be made to work

looks like they're becoming dependent on dask though https://github.com/ml6team/fondant/blob/main/components/load_from_parquet/src/main.py which is what I'd like to avoid (being dependent on any given distribution framework)

https://github.com/ml6team/fondant/blob/main/src/fondant/component.py

they're also locking themselves into kubeflow

things to consider taking from them in term of design:

• component with input and output (similar to video2dataset subsampler)
• pipeline that composes multiple components (but why is it not a component ?)
• https://fondant.ai/en/latest/#pipeline their doc is pretty clean