Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add VSICopyFileRestartable() to allow restart of upload of large files #9846

Merged
merged 3 commits into from May 15, 2024

Conversation

rouault
Copy link
Member

@rouault rouault commented May 3, 2024

Copy a source file into a target file in a way that can (potentially)
be restarted.

This function provides the possibility of efficiently restarting upload of
large files to cloud storage that implements upload in a chunked way,
such as /vsis3/ and /vsigs/.
For other destination file systems, this function may fallback to
VSICopyFile() and not provide any smart restartable implementation.

Example of a potential workflow:

char* pszOutputPayload = NULL;
int ret = VSICopyFileRestartable(pszSource, pszTarget, NULL,
                                 &pszOutputPayload, NULL, NULL, NULL);
while( ret == 1 ) // add also a limiting counter to avoid potentiall endless looping
{
    // TODO: wait for some time

    char* pszOutputPayloadNew = NULL;
    const char* pszInputPayload = pszOutputPayload;
    ret = VSICopyFileRestartable(pszSource, pszTarget, pszInputPayload,
                                 &pszOutputPayloadNew, NULL, NULL, NULL);
    VSIFree(pszOutputPayload);
    pszOutputPayload = pszOutputPayloadNew;
}
VSIFree(pszOutputPayload);

@rouault rouault added this to the 3.10.0 milestone May 3, 2024
@rouault rouault force-pushed the VSICopyFileRestartable branch 2 times, most recently from 8dea02f to 2d38652 Compare May 3, 2024 18:10
Copy a source file into a target file in a way that can (potentially)
be restarted.

This function provides the possibility of efficiently restarting upload of
large files to cloud storage that implements upload in a chunked way,
such as /vsis3/ and /vsigs/.
For other destination file systems, this function may fallback to
VSICopyFile() and not provide any smart restartable implementation.

Example of a potential workflow:
```cpp
char* pszOutputPayload = NULL;
int ret = VSICopyFileRestartable(pszSource, pszTarget, NULL,
                                 &pszOutputPayload, NULL, NULL, NULL);
while( ret == 1 ) // add also a limiting counter to avoid potentiall endless looping
{
    // TODO: wait for some time

    char* pszOutputPayloadNew = NULL;
    const char* pszInputPayload = pszOutputPayload;
    ret = VSICopyFileRestartable(pszSource, pszTarget, pszInputPayload,
                                 &pszOutputPayloadNew, NULL, NULL, NULL);
    VSIFree(pszOutputPayload);
    pszOutputPayload = pszOutputPayloadNew;
}
VSIFree(pszOutputPayload);
```
@coveralls
Copy link
Collaborator

coveralls commented May 3, 2024

Coverage Status

coverage: 69.095% (+0.01%) from 69.081%
when pulling a957dcc on rouault:VSICopyFileRestartable
into 4b1e7bf on OSGeo:master.

@rouault rouault merged commit 28be84e into OSGeo:master May 15, 2024
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants