You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to create a metaflow pipeline.
I have several steps one of those steps, I want to run in a custom docker container.
The ways I found was using either @kuberenetes decorator & specifying the image name or using @Batch decorator.
Now, since I want to test it first.
Approach I followed:
I thought of setting up minikube on my local, so for a particular step metaflow could spin up the custom container in minikube & do the process.
What went wrong:
but, while running the command.
python metaflow_test.py run --with kubernetes:image=hub.docker.com/custom_container
I get error:
Kubernetes error:
The @kubernetes decorator requires --datastore=s3 or --datastore=azure or --datastore=gs at the moment.
Conclusion:
looking at this error what I am understading is I can only spawn containers on remote k8s clusters running on cloud not on minikube.
Question:
is there a way to test locally? or should I just do the setup on cloud.
The text was updated successfully, but these errors were encountered:
What I am trying to do:
I am trying to create a metaflow pipeline.
I have several steps one of those steps, I want to run in a custom docker container.
The ways I found was using either @kuberenetes decorator & specifying the image name or using @Batch decorator.
Now, since I want to test it first.
Approach I followed:
I thought of setting up minikube on my local, so for a particular step metaflow could spin up the custom container in minikube & do the process.
What went wrong:
but, while running the command.
I get error:
Conclusion:
looking at this error what I am understading is I can only spawn containers on remote k8s clusters running on cloud not on minikube.
Question:
is there a way to test locally? or should I just do the setup on cloud.
The text was updated successfully, but these errors were encountered: