CopyFile: support assets, archives, and recursive copying

[thomas11]

This PR allows the CopyFile resource to recursively copy directories as well, similar to scp -r.

This PR enhances the CopyFile resource in a few ways.

• It can now copy directories as well, recursively with their contents.
• It now takes the standard Pulumi asserts and archives as input, allowing for seamless interop with other resources.
• It now checks whether the specified file or directory have changed (in content, not timestamp) and copies only if they did.

In light of these changes, I've renamed CopyFile to just Copy which is a breaking change.

Resolves #23
Resolves #33
Resolves #42

TODO

• The current implementation fails as soon as a file or directory already exists on the remote, like the previous CopyFile.
• Before GA, should we rename CopyFile to Copy or RemoteCopy or insert your proposal here?
• The integration test copies a bunch of EC2 setup code with TestEc2RemoteTs, which we should share instead. What would be a good way to do that, given the code is in the TS tests, not in the Go driver?

Design considerations

The behavior of the copy operation was modeled after cp and scp.

source | dest - exists as dir | dest - does not exist | dest - exists as file
-------|----------------------|-----------------------|-----------------------
dir    | dest/dir             | dest/dir              | error
dir/   | dest/x for x in dir  | dest/dir              | error
file   | dest/file            | dest                  | dest (overwritten)

Specifically:

• When copying a directory, we overwrite existing files. (The current CopyFile resource for single files does that, too.)
• When copying a directory, we don't clear the remote directory first so that no left-over files from previous copies will be around. We can always add a flag for that if needed.

Implementation notes

• I've looked around for open source Go packages implementing scp of folders, but to my surprise, I couldn't find one with an acceptable license that 1) wasn't ancient and 2) was cross-platform.
• The current implementation copies files sequentially and is therefore probably slow for large trees. If we replaced the implementation, I don't think the shape of the resource would change, so this shouldn't be a GA blocker.
• The size.ts file in both steps of the integration test is useless because it always has the same value. However, I found that without a TS file present in the additional step, it would not be run. Haven't debugged further.

[flostadler]

What if the parents of dst do not exist?

[thomas11]

When copyFile is called from our copyDir, that shouldn't be possible.

When it's called directly because the user wants to copy a file, one could argue that it should fail because the user asked to copy to a non-existant location. It's one of the questions in my (internal) ideation doc.

[iwahbe]

This looks pretty good to me.

[danielrbradley]

Why are these defined as two properties rather than just having the Asset field?

AFAIK, when a property is of type Asset, that automatically allows the user to pass in an Archive. You can see this in the SDK when your Asset fields actually get generated with the type AssetOrArchive. This whole heirarchy of types is somewhat confusing for users, but I think it makes it superfluous to include both fields.

[thomas11]

Oooh interesting! I just looked at the definitions of Asset and Archive and saw that they're separate.

[danielrbradley]

Yeah, the design of Assets overall is pretty confusing on both the provider and consumer sides 🤷

[danielrbradley]

Here's an example from azure-native:

https://github.com/pulumi/pulumi-azure-native/blob/f607b7a1e27913aef9672b75913bdf341c46c118/provider/pkg/resources/customresources/custom_storage.go#L659-L678

[danielrbradley]

This works a bit differently because it's using raw property values rather than the new framework. Perhaps this is a missing feature in the go-provider to detect which kind of asset we've got at runtime?

[danielrbradley]

I guess the question is also: what's the difference between the asset/archive types in sdk/v3/go/common/resource and sdk/v3/go/pulumi

[thomas11]

This discussion led to filing pulumi/pulumi-go-provider#237