Partitions in asset jobs created from Airbyte

[dagsir[bot]]

Hi all ,

I am trying to use partitions in my asset jobs created from Airbyte (using load_assets_from_airbyte_instance):

from datetime import datetime
from dagster import HourlyPartitionsDefinition

hourly_partitions = HourlyPartitionsDefinition(start_date=datetime(2023, 11, 3))

ingest_job = define_asset_job("source_assets", selection=source_assets_selection, partitions_def=hourly_partitions,
                              description="Pipeline to ingest/materialize all source assets",
                              tags={"code_version": os.getenv('VERSION')})

Despite of the fact that job ended , assets have been successfully materialized I cannot se any partitions.
Am I missing something or it is expected behaviour/it is not possible to create partitions/backfills when using dagster-airbyte for assets creation?

---

The question was originally asked in Dagster Slack.

[jamiedemaria]

Is source_assets_selection the return from load_assets_from_airbyte_instance?

[WojciechSzybaVeVe]

No , actually it is selection of Assets: source_assets_selection = AssetSelection.groups("mysql") | AssetSelection.groups("dynamodb") where both groups have been created using load_assets_from_airbyte_instance

[WojciechSzybaVeVe]

I think the problem is that load_assets_from_airbyte_instance not allows to pass partition_def to _build_airbyte_assets_from_metadata within assets,defs.py. When I've added it there manually in multi_asset decorator it is working perfectly

[WojciechSzybaVeVe]

I've tried to use with_resources to add this to the context but it also seems not to be working with load_assets_from_airbyte_instance and selection of assets

[jamiedemaria]

yeah there isn't a parameter to add a partitions definition in load_assets_from_airbyte_instance, which tells me that airbyte assets don't support partitions. @benpankow can you confirm this? Is adding partition support to airbyte assets something we should consider?