View metadata for each partition in Global Assets Lineage

[elliottower]

What's the use case?

I am using partitioned assets to process an umber of files, for each file, it is handy to be able to click through and see metadata (e.g., preview, number of rows, etc). However, with partitioned assets you pretty much always have to click through to the assets catalog to see anything useful.

Assets view:

Ideas of implementation

In the current implementation, it lists the names of the partitions, instead of this it would be awesome if you could click each of the partition names and at least see metadata and tags there.

The assets view also has a link to each run, which you currently can't get from the Global Assets Lineage screen, maybe that could be added as well for each partition.

Currently partitions make things a bit difficult because you can't observe all the metadata as easily with a global lineage view, adding the ability to drill down and view metadata per partition would be awesome.

I believe currently, when there's only one item in a partition, it will show the metadata and tags from that item. In regular partitions with multiple items, even if it showed the first item of the partition's info like with single case, it would be helpful. Ideally it would let you see them all though.

In the assets view, in the overview, it also does the same where it shows metadata from only one of the partition items (see image below), which is better than nothing. You have to click through to partitions to view more, which is fine. Basically it would be nice to have similar functionality to this partitions view in the global assets lineage.

Edit:

It would also be nice to see when each partition was materialized, as it currently just shows 'materialized' but when you start a backfill, it will keep saying that for a few seconds until it updates and says in progress. But if it's a quick backfill, you may miss that time window, and you just see the green materialized text again. You have to go through to the assets view to really do anything.

With regular assets (screenshot from dagster university), it shows time of materialization and has a link to the run if you click it. Maybe it could at least show the most recent timestamp of materialization, and click through to the partitions page? Not sure but currently you lose a lot of the nice UI features when you use partitions when compared to regular assets

Additional information

(I edited the screenshots to remove the names of partitions because they are sensitive info, sorry)

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.