Setting data loader config without JSON file

[PCanavelli]

Hey @bghira

What would you say is the best way to set the data loader config directly through Python, without relying on a JSON file?

Doing it for the trainer itself is straightforward with something like:

from Simpletuner.helpers.configuration.json_file import normalize_args

training_config = {
    "--data_backend_config": str(data_loader_config_path),
    "--resume_from_checkpoint": "latest",
    ...
    "--validation_torch_compile": False,
}

data_loader_images_config = {
    "id": run_id,
    "type": "local",
    ...
    "metadata_backend": "json",
}

data_loader_text_config = {
    "id": "text-embeds",
    ...
    "write_batch_size": 128,
}

data_loader_config = [data_loader_images_config, data_loader_text_config]

Trainer(normalize_args(self.training_config))

But I'm finding it a lot less obvious where and how to set data_loader_config to do the same for the data loader.

[bghira]

we just write it to disk, even when using simpletuner via FastAPI.