Pickled assets on serverless are not accessible

[gnilrets]

We're using dagster serverless. I developed an asset pipeline locally and got everything working. I have a few assets that return some small simple Python classes that are then used as inputs in other assets. These assets are just pointers to Snowflake objects. Locally, I could materialize the assets independently without issue. However, in serverless, it looks like the materialized assets are not accessible because they're pickled in a /tmp directory.

What is the right/easy way to persist these simple objects in serverless? Do I need to set up a cloud storage integration or is there something easier for small materialized assets?

[prha]

Are you binding an IO manager explicitly? The IO manager in serverless should default to a cloud-backed serverless specific IO manager rather than the local filesystem-based one.

If you are explicitly binding an IO manager to your jobs, you may want to bind it to the following:

from dagster_cloud.serverless.io_manager import serverless_io_manager

...
defs = Definitions(
    ...
    resources={"io_manager": serverless_io_manager}
)

[prha]

@gnilrets are you using PEX, or the fast deploy CI/CD integration? I'm wondering if there's some early binding happening when we build the PEX image. We could confirm that if you see the serverless io manager when you turn off fast deploys.

[gnilrets]

@prha - Do you think that the deployment method would affect the io manager for dbt cloud? That seems to be what's messing things up here. If I remove dbt cloud job from my defs, it uses the serverless io manager by default. When I include the dbt cloud job, it uses the filesystem io manager.

Either way, I don't think I'm using PEX because I've never heard of that before. I'm building a custom image and deploying as follows (in gitlab CI):

    # Build the dagster cloud container
    - DOCKER_BUILDKIT=1 dagster-cloud serverless build
      --base-image=eta-dagster-app
      --deployment=${DEPLOYMENT_ID}
      --source-directory=.
      ${CI_COMMIT_REF_NAME}

    # Get dagster cloud registry info
    - dagster-cloud serverless registry-info
        --organization=${DAGSTER_CLOUD_ORGANIZATION}
        --api-token=${DAGSTER_CLOUD_API_TOKEN}
        --deployment=${DEPLOYMENT_ID}
      > registry_info.env
    - source registry_info.env

    - dagster-cloud workspace add-location
      --organization=${DAGSTER_CLOUD_ORGANIZATION}
      --api-token=${DAGSTER_CLOUD_API_TOKEN}
      --deployment=${DEPLOYMENT_ID}
      --package-name=eta_wh
      --image=${REGISTRY_URL}:${CI_COMMIT_REF_NAME}
      mainspring_warehouse

[prha]

@gnilrets I've tried adding dbt cloud assets to a test project on serverless and still am not able to repro. I'm wondering if the culprit might be
A) if there's something specific about how you're building your code location images
B) there's some bug in the cacheable dbt cloud assets (are you using cacheable job definitions as outlined in https://docs.dagster.io/integrations/dbt-cloud#step-4-cache-the-dbt-cloud-job-compilation?).

For A) how are you uploading the built image (e.g. before calling the add-location command? The latest versions of the gitlab CI script makes uses of the dagster-cloud serverless deploy command: https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/serverless-ci.yml

For B) If you set up job caching, I wonder if it makes a difference to manually set the DAGSTER_DEFAULT_IO_MANAGER_MODULE / DAGSTER_DEFAULT_IO_MANAGER_ATTRIBUTE variables in your CI environment.

[gnilrets]

Oh hey! I was was changing too many things at once. I think I've finally pinpointed the issue. The dbt cloud thing was just a red herring.

It was just the fact that I didn't have the boto3 library. If I reset back to where I started and just add that package in my requirements file, it correctly uses the local filesystem io manager when run locally, and the serverless one when run in serverless.

I guess the only thing here to do is that maybe boto3 should be an explicit dependency of the dagster-cloud package.

Thanks for your help @prha !

[prha]

Oh, great. Glad you figured it out! I think our documentation suggests that you have dagster-cloud[serverless] in your dependencies, which includes boto3 (should have spotted that earlier).

[gnilrets]

Got it, I suppose this is just the pain of being an early adopter when the product continues to evolve....