feat: add scalar kind

[jedevc]

DEV-3403

Fixes #6466

This is the "proper" implementation of #6739. This allows us to preserve the original scalar name.

Implemented functionality:

• TypeDefs have an AsScalar field, so that they can be connected to their original value.
  • For example, this allows the CLI to know that a flag is a dagger.Platform flag. Eventually, this could allow us to have a special value, like "current" as a shorthand to represent the client platform.
• Modules can call WithScalar to define their own non-core scalar values - these are represented as scalars in TypeDefs, as well as in the GraphQL schema.
• Other modules can import another module that have a custom scalar, and use it.

TODO:

• Implement all the above functionality
• Add a magical CLI handler for dagger.Platform (this was my motivation for doing all this lol)
• Add some tests
• Fixup all TODOs

[jedevc]

A fun side-effect I just discovered - because of how dagql does enums, we get enum support included alongside this!

[jedevc]

Note: this is only fully enabled for the Go SDK right now - there will need to be minor updates to both Typescript and Python (cc @TomChv @helderco), so that functions that take a scalar argument mark this using WithScalar to the engine (instead of using WithString).

[helderco]

I'll get on that asap and coordinate with @TomChv. Thanks for pushing this through! 🙏

[wingyplus]

Asking for #6967, is WithString still support?.

[jedevc]

Asking for #6967, is WithString still support?.

@wingyplus, yes, WithString is still here - it's just that now, by using WithScalar for a type like Platform, it indicates it as specifically a Platform input, instead of just a normal string - this allows the CLI to have a significantly better interface for it/etc.

[jedevc]

@helderco should be ready for another review now - do we want to land the typescript/python work in the context of this PR, or do we want to follow-up in later PRs?

[helderco]

It's more important to be in the same release than in the same PR. I've already made the change for Python locally but in holiday today and the tests need improvement. I asked Tom to do the same but it may not be as easy there because of how it's written. Hold on at least for Python and I'll check up with Tom.

[jedevc]

Rebased, since main is a little less flaky right now 🎉

[TomChv]

@jedevc @helderco

I pushed the support for TypeScript on your PR directly: a0a9d61

Both Scalar(a0a9d61) and Enums(bf11e82) are tested

[helderco]

@jedevc, needs a change log. SDKs not needed, I think. Just core.

[jedevc]

@helderco will do - looks like TestModulePythonScalarKind and TestModulePythonEnumKind are failing in CI though.

[helderco]

@helderco will do - looks like TestModulePythonScalarKind and TestModulePythonEnumKind are failing in CI though.

Ah, forgot about those. Will fix in a minute 🙏

[helderco]

@jedevc, user created enums are actually very useful to support (instead of custom scalars):

• ✨ functions: support enum argument types #6780

[TomChv]

user created enums are actually very useful to support (instead of custom scalars):

This implies some changes in the TS SDK to support enumeration but it shouldn't be a big deal.

[jedevc]

Yeah this is actually a bit more of a pain in go since native enum types aren't really a thing. I'm happy for someone else to lead on the custom enums work and I can work out how it makes sense in go, but probably makes sense to lead with a language that actually supports them nicely 😆

See main...jedevc:dagger:add-custom-scalar-kind for a base for how to handle the custom scalar plumbing (enums are pretty similar to enums for this).

[helderco]

I think Go's implementation of enums is fine. Custom scalars are more problematic because their implementation has to define:

• How the scalar's value is represented in the module's language (aka, back-end representation)
• How the value's back-end representation is serialized to a JSON-compatible type
• How the JSON-compatible representation is deserialized to the back-end representation

That can be simplified if we just assume/limit to a normal string everywhere, but it still opens a can of worms when we don't have enough feedback that it's actually needed.

However with enums, it's just a fixed list of supported string values. No serialization/deserialization required. To support defining new enums via the API it’s similar to ObjectTypeDef, where the fields are just a list of values and their descriptions:

type EnumTypeDef {
  name: String!
  description: String!
  values: [EnumValueTypeDef!]!
}

type EnumValueTypeDef {
  name: String!
  description: String!
}

type TypeDef {
  """
  If kind is ENUM_KIND, the enum-specific type definition.
  If kind is not ENUM_KIND, this will be null.
  """
  asEnum: EnumTypeDef
    
  """
  Returns a TypeDef of kind Enum with the provided name.
  
  Note that an enum's values may be omitted if the intent is only to refer to an enum.
  This is how functions are able to return their own, or any other circular reference.
  """
  withEnum(name: String!, description: String = ""): TypeDef!

  """
  Adds a static value for an Enum TypeDef, failing if the type is not an enum.
  """
  withValue(name: String!, description: String = ""): TypeDef!
}

enum TypeDefKind {
  """
  A GraphQL enum type and its values.
  
  Always paired with an EnumTypeDef.
  """
  ENUM_KIND
}