One asset graph with multi simultaneous jobs with different config_schemas

[rptechcxo]

I created an asset graph that processes similar data for various clients. The plan was to use the config_schema feature to create multiple jobs that would run simultaneously for each client. The issue I ran into is that the runs are not isolated. The data for a given tree is always saved in the same location, i.e., path\to\dagster_home\storage\asset_name. The jobs overwrite each other if they run simultaneously. I tried using the docker containers to isolate runs, but that didn't work, because it still saves to the same location.

Is this idea I'm trying to implement not possible as I've envisioned it? Is there a way to isolate job outputs if they use the same graph tree?

I'm trying to determine if I need to rebuild my process.

Thanks,

[rptechcxo]

I think combining the graph-based assets and op factory is going to be my solution, with some additional changes to the architecture.

[dmarteau]

I bumped into the same issue, can you elaborate your solution, thx ?

[rptechcxo]

Well the idea mentioned above only solved part of my problem. If you look up op factory in the documentation you see how you can dynamically create assets when you startup your repository. This replaces the config_schema part. The problem is it will clutter the side panel (not necessarily a big deal) and the number of asset trees/jobs is fixed after startup. If you ever want to add another asset tree you have to restart the repository.

The additional changes in the architecture was referring to using mongo, sql and/or aws sqs to store intermediate steps instead of dagster artifact storage.