Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return full failsafe path programmatically as part of truncate and load #43

Open
jacobdadams opened this issue Feb 28, 2023 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@jacobdadams
Copy link
Member

Cloud functions can't mount cloud storage as a file system, so we'd have to pass a temp directory as the failsafe_dir, get the full path programmatically, and copy that over to a bucket.

@jacobdadams jacobdadams added the enhancement New feature or request label Feb 28, 2023
@jacobdadams jacobdadams added this to the v3.1 milestone Feb 28, 2023
@steveoh
Copy link
Member

steveoh commented Feb 28, 2023

def save_feature_layer_to_json(feature_layer, directory):
    """Save a feature_layer to directory for safety as {layer name}_{todays date}.json
    Args:
        feature_layer (arcgis.features.FeatureLayer): The FeatureLayer object to save to disk.
        directory (str or Path): The directory to save the data to.
    Returns:
        Path: The full path to the output file, named with the layer name and today's date.
    """

The cloud storage bucket or blob name could be passed here and the dataframe json could be written directly to a storage bucket if the directory follows a naming convention. It's redundant to write data to disk only to then read it and write the same data to cloud storage.

Here's an example using the parquet format from pandas being uploaded to cloud storage in memory.

https://github.com/agrc/udot-parcel-ml/blob/af3c04e142af74520e2e481c1b8f31769fc3cc41/row.py#L749-L752

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants