Use local adaptors in a local Lightning instance

[amberrignell]

User story

As a user that has built a new adaptor locally, I would like to spin up Lightning and use that adaptor, so that I can see it in action and properly test it.

Details

Implementation notes

Release notes

User acceptance criteria

[josephjclark]

Here is a design for this:

When Lightning populates the adaptor list, it should do some from two possible sources: npm or the monorepo.

If Lightning detects the OPENFN_ADAPTORS_REPO env var (which is is a path to the repo and is used by the CLI to interface with the monorepo), it should enable the monorepo source.

Any adaptor version found in the monorepo should be displayed at the TOP of the picklist for new adaptors, with version set to local. This internally needs to set the version on the Run to @local (ie, @openfn/language-common@local). That is, the pick list label should be local and the value should be @openfn/language-common@local.

The monorepo source needs to be populated by looking in the packages/ folder of the adaptors monorepo. Every folder inside packages is an adaptor which should be added to the list of options.

In order to support this, the Worker will need to have special handling for the @local version. Not a terribly big deal, see OpenFn/kit#806

[josephjclark]

@stuartc I have (almost) implemented support for local/monorepo adaptors in the worker in #808

tl;dr: if lightning sets the adaptor version to @local on a given step, the worker will try to load it from the monorepo

This has been useful for collections so I just ploughed on with it

Edit: that's merged. So all we need to do is update the picklist with an @local option and we can support local adaptor dev in lightning

[christad92]

• Update the adaptor picklist with an @local option and we can support local adaptor dev in lightning

[josephjclark]

Hey, I've been thinking about a different (cheaper) design for this

Maybe we can start the worker in a mode that ONLY uses the monorepo to load adaptors. Driven by an env var maybe.

So if you do:

LOCAL_ADAPTORS=true mix phx.server

Lightning will start the worker with a flag that forces all adaptors to be local.

The advantage is that the dev effort is much less. No need to faff around with this @local stuff. It is ARGUABLY better for users to - suddenly everything runs out of your monorepo no matter what version is set in your workflow. You can argue the toss but maybe it's a cleaner experience. More predictable.

Counter-argument to that: if you're running with the monorepo you have to remember to keep your local build up to date. It's very easy to forget to build and run an old version of an adaptor by accident. That's why I like the @Local approach - it's more faff (for us and for users) but it's very explicit.

The disadvantage is that Lightning is lying to users. The workflow will say like "this step uses http 6.0", but actually the runtime log will say "I'm using whatever is in your monorepo". That's kind of confusing and maybe a bit tricky. And compounded by the problem of forgetting to rebuild. Suddenly it's not actually clear what version of the adaptor you're even running.

I suppose we could do something like: if the LOCAL_ADAPTORS env var is set, we don't show a version picklist in the UI. That's a bit more honest.

Maybe the question is: for a developer using lightning to test adaptors, do I want to set an individual job to use a local adaptor, or do I just want to always use local adaptors?

[midigofrank]

Hmm @josephjclark I actually think the OPENFN_ADAPTORS_REPO approach is much straightforward and simpler.

LOCAL_ADAPTORS=true introduces an "operating mode" in the app. This might be simpler at the beginning but will become costly when we want to introduce something new which affects the adaptors. With OPENFN_ADAPTORS_REPO, all we have to do is update the adaptor manifest

Here is a design for this:

When Lightning populates the adaptor list, it should do some from two possible sources: npm or the monorepo.

If Lightning detects the OPENFN_ADAPTORS_REPO env var (which is is a path to the repo and is used by the CLI to interface with the monorepo), it should enable the monorepo source.

Any adaptor version found in the monorepo should be displayed at the TOP of the picklist for new adaptors, with version set to local. This internally needs to set the version on the Run to @local (ie, @openfn/language-common@local). That is, the pick list label should be local and the value should be @openfn/language-common@local.

The monorepo source needs to be populated by looking in the packages/ folder of the adaptors monorepo. Every folder inside packages is an adaptor which should be added to the list of options.

In order to support this, the Worker will need to have special handling for the @local version. Not a terribly big deal, see OpenFn/kit#806

[josephjclark]

I had a random moment of clarity on this late last night!

As a user, I am pretty sure that I want local adaptors to be a global setting. Run ALL my adaptors from the monorepo, or run NONE of them. It's not worth the effort to keep switching individual jobs to the @Local version.

Also remember that if I set my local step to use @Local, at some point presumably I have to set it back again.

The CLI works the same way: I run the CLI in "local monorepo mode", which makes ALL adaptors in my workflow load from the monorepo. I think it should be exactly the same in Lightning.

Yes, that does mean that when you run in this mode you need to remember to build your monorepo. But you also have to explicitly put Lightning in this mode with a startup flag - I think it's quite reasonable.

So I strongly suggest the design is:

• Set an env var eg LOCAL_ADAPTORS=true to put Lightning AND the worker into Monorepo Mode. Or maybe it's a flag, not an env var. I don't have a strong opinion - a flag is probably more intentional.
• In Monorepo mode, we don't show adaptor versions in the picklist in the workflow editor (and probably at the top of the inspector too?). We force all versions to the string local. This gives the user important feedback.
• In Monorepo mode, we either a) pass a flag to the worker to force all adaptors to run locally (needs a fix in the worker, quite easy I think), or b) when assembling Run objects, we set the adaptor version to @Local before sending it to the worker. Probably a) is better.
• Ensure the OPENFN_ADAPTORS_REPO env var is set if Monorepo Mode is true, and refuse to start if it is not

[josephjclark]

Random thought on this!

We need to bump the local worker version included in assets/package.json. Aside from some other fixes, we'll probably need to include a worker patch with local support. So this is just a reminder to bump the version.

[midigofrank]

Hey @josephjclark FYI, I have opted to send in the adaptor with @local, at least for now. 2 things I am battling with:

1. When building the adaptor list, should we merge in both the local ones + npm ones. At the moment, I'm picking one or the other (depending on the flag) and I'm running into a problem when editing an existing job. If you switch to local adaptors only and the job was using an adaptor that doesn't exist locally, the select field doesn't show it.
2. How do we load docs from local adaptors?

[josephjclark]

I think you're right to use npm OR local adaptors, not both. What adaptors exist that aren't in the local monorepo? I'm probably happy to accept in error in these rare cases.

Good point on docs - but it's probably OK to ignore them for now. This is a very dev facing feature. We want local adaptors to test new APIs and bugfixes, not to browse the docs.

This doesn't need to be bulletproof or user friendly. It's for Mutchi and Hunter to test adaptor fixes in context. It's OK for it to be rough (especially right now)