Skip to content

Commit

Permalink
Fix an issue with get_cards not respecting a Task's ds-root (Netflix#…
Browse files Browse the repository at this point in the history
…1111)

get_cards would not always respect a Task's ds-root leading to cases
where a Task has cards but they cannot be accessed because of an invalid
path.
  • Loading branch information
romain-intel authored Aug 18, 2022
1 parent c2d615e commit eda2f2d
Showing 1 changed file with 8 additions and 14 deletions.
22 changes: 8 additions & 14 deletions metaflow/plugins/cards/card_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -261,24 +261,18 @@ def _get_flow_datastore(task):
# We need to set the correct datastore root here so that
# we can ensure the the card client picks up the correct path to the cards

for meta in task.metadata:
if meta.name == "ds-type":
ds_type = meta.value
break

ds_root = CardDatastore.get_storage_root(ds_type)

if ds_root is None:
for meta in task.metadata:
# Incase METAFLOW_CARD_S3ROOT and METAFLOW_DATASTORE_SYSROOT_S3 are absent
# then construct the default path for METAFLOW_CARD_S3ROOT from ds-root metadata
if meta.name == "ds-root":
ds_root = os.path.join(meta.value, DATASTORE_CARD_SUFFIX)
break
meta_dict = task.metadata_dict
ds_type = meta_dict.get("ds-type", None)

if ds_type is None:
raise UnresolvableDatastoreException(task)

ds_root = meta_dict.get("ds-root", None)
if ds_root:
ds_root = os.path.join(ds_root, DATASTORE_CARD_SUFFIX)
else:
ds_root = CardDatastore.get_storage_root(ds_type)

storage_impl = DATASTORES[ds_type]
return FlowDataStore(
flow_name=flow_name,
Expand Down

0 comments on commit eda2f2d

Please sign in to comment.