implement programmatic default providers in the engine and Go SDK

[tgummerer]

Implement default providers in the engine and the Go SDK first (Go SDK for no other reason other than I'm most familiar with that language).

The SDK sends a RegisterDefaultProvider request to the engine, which adds the provider to its default provider map, and explicitly disallows creating an implicit default provider going forward.

The locking around this happens on the SDK side, since we need to disallow RegisterResource requests happening at the same time as DefaultProvider requests. Here we can simply take a read lock before creating the RegisterResource goroutine, and a write lock for the duration of the RegisterDefaultProvider call.

This way we can have RegisterResource calls happen in parallel, but the RegisterDefaultProvider call will not go ahead before the read lock is released. Similarly, the next RegisterResource call will be locked until the RegisterDefaultProvider call has finished.

/xref #2059

This could benefit from a conformance test, however we do not have them for Go yet, and making them work also means implementing the changes in the Python and NodeJS SDKs. I was planning to do that in a separate PR to split the changes up a little bit, but can add more commits here if that's preferred.

Checklist

• I have run make tidy to update any new dependencies
• I have run make lint to verify my code passes the lint check
  • I have formatted my code using gofumpt

• I have added tests that prove my fix is effective or that my feature works

• I have run make changelog and committed the changelog/pending/<file> documenting my change

• Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

[pulumi-bot]

Changelog

[uncommitted] (2024-05-06)

Features

• [sdk/go] Allow default providers to be set programmatically
  #16105

[EronWright]

I think this is a critical aspect. I observe that the default provider may or may not have a version component, depending on which SDK is in use. I would argue that there should be one default provider per package, not per package+version.

See related: #16109

[tgummerer]

I think it needs to be per package+version. Since users can pass in the version on resource registration, the following would not work otherwise:

		_, err = random.NewRandomPet(ctx, "example", nil, pulumi.Version("4.2.0"))
		if err != nil {
			return err
		}
		_, err = random.NewRandomPet(ctx, "example2", nil, pulumi.Version("4.3.0"))
		if err != nil {
			return err
		}

[Frassle]

I would argue that there should be one default provider per package

Thomas is correct, this definitely needs to be per pkg+version. The fixes in #16109 remove the "unversioned" default provider, and that might be ok as well, need to check it doesn't break anything odd backcompat wise.

[EronWright]

Is provider a urn, can the comment say that?

[Frassle]

It should be a 'ref' of the form "::"

[Frassle]

It should be a 'ref' of the form "::"

[Frassle]

I don't think this is needed? The engine can work everything out from the provider instance it has in memory.

[tgummerer]

There's a couple of problems here that I was trying to solve with adding the version and pluginDownloadURL fields:

• I don't think the provider in the engine saves the pluginDownloadURL anywhere, so we'll have to pass that along, as it's part of the defaultprovider map key
• Once the provider is constructed in the engine, it will always have a version. So there's no way to tell whether it was constructed with or without a version in the SDK. Now this is very closely related to the TODO Eron also commented on implement programmatic default providers in the engine and Go SDK #16105 (comment). If we decide that we should at least also always set the default provider for where the key is just pkg, I don't think we need to send the version along here, as we can just set d.providers[pkg] and d.providers[pkg-<version>]. Without sending the version field along here we would only set d.providers[pkg-<version>].

[Frassle]

I don't think the provider in the engine saves the pluginDownloadURL anywhere, so we'll have to pass that along, as it's part of the defaultprovider map key

It should be on the provider input state.

If we decide that we should at least also always set the default provider for where the key is just pkg

We shouldn't! But if you get given a "ref" that points to a specific provider which will have a specific version (again on its input state). We're just going to be using that provider instance as the default for it's matching pkg-version.

[tgummerer]

Right, we'll need #16109 for that, but then I think it works.

[EronWright]

Since defaultProviders seems to allow side-by-side versions, why aren't the explicit providers handled in this same way?

[tgummerer]

Since defaultProviders seems to allow side-by-side versions, why aren't the explicit providers handled in this same way?

They are. We do allow side-by-side versions here, and I think that's the right thing to do as well. We only disallow the creation of new internally created default providers with different versions than the programmatically set default providers. The idea is that otherwise it's easy for users to set a default providers, and then maybe pass a Version option into one of the resource creation calls, and then accidentally create a new internally created default provider through that, which would be a bit confusing imo.