Change how Meteor executes Async code #11505
Replies: 57 comments 85 replies
-
|
Thanks a lot for this overview @filipenevola ! |
Beta Was this translation helpful? Give feedback.
-
|
I'm working in a few changes to be possible to use mongo package already using the standard Node.js async at least in the API. I'll update this discussion when I have code ready to be shared. The idea of this first phase is to provide the API in a way that removing Fibers later the new API will be exactly the same so developers don't need to migrate the mongo use cases they are going to add from this point forward. I'm still trying different ideas for the final API so I can't share much at this point but my ideal API would be something like: // brainstorm :) we don't know the final API yet
const LinksCollection = await createCollection('links');
const links = await LinksCollection.find();In this process I'm finding a few challenges that I would like to describe at least one briefly here. For example, we can't just change all the functions to async functions because in many cases the function is used more like a class, creating new instances using const collectionDefinition = new Mongo.Collection('links');
const LinksCollection = await collectionDefinition.asyncInit();I'm not mentioning here the natural challenge of converting to That is all for now, the motivation of this message is just to provide more visibility for the community. |
Beta Was this translation helpful? Give feedback.
-
|
My main concern is how this will affect tracker, method stubs, and any other api's used on the client that requires knowing the context code is run within, since the current way it does that requires the code to be synchronous. Does one of the parts of the plan include looking into this?
This gave us the equivalent of top-level await where we could have async code (using fibers) in the top-level of a file, and Meteor would wait for it to finish before loading additional files. Do we have any plans to implement top-level await in Meteor's module system? |
Beta Was this translation helpful? Give feedback.
-
Here are a couple examples of context on the client:
Being able to run async code in the top-level of a module is more convenient in some cases and gives more control over when code is run. It isn't needed for Meteor apps to work, but is something we would lose from removing Fibers unless we implement an alternative. The ECMAScript proposal for top-level await is stage 4, so it should be part of the next version of ECMAScript, but we will have to implement it in our module system since we don't use native ECMAScript modules. |
Beta Was this translation helpful? Give feedback.
-
I don't see why the changes in the server would affect client side, besides the API to keep the code compatible with server and client. Could you explain why? Even if the code has a few async functions we can always await these calls. If your concern is the server, we will use |
Beta Was this translation helpful? Give feedback.
-
Yes, I'm following this proposal as well. It would be nice to support it as we could provide better APIs but I don't see it as required but definitely a nice to have. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure this is an option. On the client we don't have something like AsyncLocalStorage, so the execution context is always global. This works well when the code is synchronous, but not when it is async. If the execution context is reset after the promise resolves, Meteor would have to be very careful about other code running at the same time. For example, while running an async method stub, if a tracker computation or click event listener called collection.insert, it would cause problems if Meteor thought that insert happened in the method stub. There is one option I've seen to having execution context with async code - we can use zones.js or a similar technique to it where it wraps all native api's that can cause code to run later, though there are a few downsides to doing something like this. There are places that use tracker and minimongo where it might not be feasible or desirable to make it async. For example, blaze helpers, or most of the existing tracker integrations with the different view layers. Also, making tracker async might have interesting effects on when Tracker computations are re-run that might need to be considered.
I agree. |
Beta Was this translation helpful? Give feedback.
-
|
@zodern I concur with your remarks on minimongo / tracker impacts. Can it not be a tought to only address Meteor calls and it stubs and leave alone other Mongo calls on client side. That would make refactoring effort at least an order of magnitude lighter. |
Beta Was this translation helpful? Give feedback.
-
|
After hitting the wall this morning with async and Meteor 2.3 (thanks @filipenevola for the links!) this topic really catched my attention.
As this is all a lot of work, I was wondering if before building these new packages, a modern "skeleton" for these "async" packages could be defined:
I'm not sure if you actually run into these problems at all in practice, new Mongo.Collection doesn't have to do anything asynchronously at all, so this will be perfectly possible I guess? |
Beta Was this translation helpful? Give feedback.
-
We are going to decide details together along the way but yes, this is the idea, to provide a new skeleton Fiber-free and also provide a way for people with existent projects to write new code that doesn't depend on Fibers, first at least in the API level. So no migration would be necessary at least for the newly added code. I'm not sure if it would be a good idea to also create npm packages in the same process as it could add more challenges that are not really necessary at this point.
Actually Mongo.Collection constructor does have some async calls inside it as you can see here and indirectly here. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for interrupting this discussion. Is this package helpful as an alternative? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @JoHuang I was checking this package and it could be a nice alternative to Fibers at least to avoid async layers but it seems (by looking to the issues) that this package also has issues with other Node.js features. Are there any large projects like Meteor using this package? |
Beta Was this translation helpful? Give feedback.
-
|
For my application code, I assume this means that pretty much all calls to the Mongo and Accounts package will become async calls instead. The idea above to create alternate packages with new api signatures (e.g. mongo-async) sounds promising in theory but I wonder what the ripple effects will be. Should every meteor package that interacts with mongo or other outside services create an alternate "async" package with new signatures and then piece by piece move the implementation from the "sync" (deprecated) version to the async version? Below are the obvious ones I could see that I use. Some will probably need forking since they have been abandoned.
Just trying to wrap my head around the order of execution here outside of the core meteor packages |
Beta Was this translation helpful? Give feedback.
-
|
Hi @perbergland I'm not sure if I understood this sentence correctly:
Are you saying that we could avoid this work by using About the changes to async calls in general, yes, that is probably going to happen in the packages listed by you. It's important to understand that this was the chosen direction by TC39 so we don't have other options if we want to follow ECMAScript standards as close as possible. This is the way Javascript projects should write code nowadays, with async layers. About the packages, it would be nice to also keep a list of Community packages that are going to be affected by these changes. |
Beta Was this translation helpful? Give feedback.
-
No, I was just trying to think of the progression of work here. Let’s take Collection.findOne as an example. Its signature is findOne(selector?: Selector<T> | ObjectID | string): U | undefined;Phase one is to introduce its sibling in Mongo-async with an async signature: findOne(selector?: Selector<T> | ObjectID | string): Promise<U | undefined>;The implementation in phase 1 could be as easy as just invoking the sync method in the -async package. The way I understand it, phase 2 would be to move the real implementation chunk by chunk into the -async package and then the original package implementation would become a shim on top of the async package where the sync method is just invoking findOne(...args) {
return findOneAsync(...args).await();
}I think we’re going to need to come up with good patterns here that all packages can use (and probably helper methods within Meteor to make it possible to standardize how deprecation warnings are emitted also from packages outside Meteor core). Note: The code in meteor/mongo is pretty complicated with write fences and what not, and may not lend itself to being converted in this way. It was more for getting the discussion to a level where we start talking in code examples. Using Typescript as the language for this is a natural choice, I think. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the general idea is exactly what you said, specially for the first step but there are some quirks as with Fibers we can pause the execution even inside constructors. This is hard with async functions as a constructor is always going to return a new instance of the desired class and not a Talking about This is important so people can continue to develop new code and this new code is not going to suffer rewrites in the future. A few examples that I'm using while I'm making these changes to // api/LinksCollectionAsync.js
import { createAsyncCollection } from "meteor/mongo";
export const LinksCollectionAsync = createAsyncCollection("links");// api/linksInsert.js
import { Meteor } from "meteor/meteor";
import { LinksCollectionAsync } from "./LinksCollectionAsync";
export const insertLink = async ({ title, url }) =>
LinksCollectionAsync.insertAsync({
title,
url,
createdAt: new Date()
});
Meteor.methods({
async insertAsync(arg) {
await insertLink(arg);
}
});// server/main.js
import { Meteor } from "meteor/meteor";
import { LinksCollectionAsync } from "../api/LinksCollectionAsync";
import { insertLink } from "../api/linksInsert";
Meteor.publish("linksAsync", async () => LinksCollectionAsync.findAsync());
Meteor.startup(async () => {
const linksCursor = await LinksCollectionAsync.findAsync();
if (linksCursor.count()) {
console.log(`No need to insert the first link`);
console.log(`Links: `, linksCursor.fetch());
return;
}
await insertLink({
title: "Do the Tutorial",
url: "https://www.meteor.com/tutorials/react/creating-an-app"
});
console.log(`Links: `, linksCursor.fetch());
});The goal is also to be able to have the same collection with two instances, one that is I'll keep updating this discussion as I go further and before making final decisions I'm going to open a new discussion specially for this first step on Everything that is being said in this discussion is being considered and will be considered in the future as well. |
Beta Was this translation helpful? Give feedback.
-
Is there code in the mongo package where this is actual a problem? Wouldn't this be simply wrapped in a factory function, like |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jankapunkt, this is basically what I'm doing, of course, it is not that simple but the idea is the same. |
Beta Was this translation helpful? Give feedback.
-
|
I just published a draft PR for Mongo Package Async API #11605 |
Beta Was this translation helpful? Give feedback.
-
|
@filipenevola, I was happy to watch your Meteor Impact talk about returning to the roots and - while I understand the arguments - I feel like eventual remove of If there is no other future-proof solution except let x = 0
const inc = () => { x += 1 }into this (pseudocode) let x = 0
const inc = () => { x += 1; updateDOM() }I don't know to what extent it's technically possible, but maybe Meteor could track the usage of its exposed async API, like Mongo operations, and add Generally, I think that adopting this approach could be a huge step in the "less code, more power" direction you mentioned in your talk. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @red-meadow I understand your concern but what we are realizing is that we need to be as much aligned as possible with the Node.js ecosystem as a whole. This is important to keep Meteor familiar for Node.js developers and vice-versa. Supporting standard async and await is important in this direction, especially as Fibers (coroutines) is not the way Node.js decided to move forward with async execution. Meteor is not relying a lot on Fibers at the API level so we believe the changes are not going to affect users a lot but in the end, it's going to be much easier to integrate Meteor code with Node.js code in general. |
Beta Was this translation helpful? Give feedback.
-
|
Any progress on this issue? We are getting more and more people asking for MeteorJS support from APM and are looking forward to support it as soon as the switch to async/await is done. |
Beta Was this translation helpful? Give feedback.
-
|
@filipenevola is there a plan to support the old fiber sync style API for a certain number of minor versions or will it be breaking, once the transition is complete? |
Beta Was this translation helpful? Give feedback.
-
|
The idea is in Phase 2 to have it as an option:
And only in Phase 4 it would be required not to use Fibers:
What is your concern specifically? The time between people finishing the code migration and Meteor releasing new versions? We will still support the Fibers versions for essential fixes and improvements for a reasonable amount of time, except when these are very hard or impossible to do in the Fibers version. |
Beta Was this translation helpful? Give feedback.
-
|
@filipenevola it feels like there should be a Phase 5 in the list, which would be 'Dropping support for fibers'. Presumably to be done when meteor wants to update to node 16, assuming Phase 1.1 isnt completed. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Julusian when Phase 3 is completed we will be able to run Meteor with Node.js 16.
I'm not sure if we need an explicit Phase 5 as we state in Phase 4 that everything that uses Fibers is going to be in maintenance mode. So in newer versions of Meteor supporting Node.js 16, for example, Fibers is not going to be there anymore. |
Beta Was this translation helpful? Give feedback.
-
|
In my experience 'maintenance mode' is generally when it will still be supported, with the only code changes being bug fixes. So it is unclear as to when fibers will stop working in an up to date version of meteor.
Based on this comment, it sounds as though they also find it unclear in what version fibers will be not at all supported |
Beta Was this translation helpful? Give feedback.
-
|
I’d be happy to help migrate the dependent code, esp all the code in accounts_base, but I’m unsure how to convert all methods that invoke mongo methods to be async without the IDE support that e.g. moving to Typescript would give. Or are there tools that help detect invocations that lack the needed await? As an example, take destroyToken in accounts_server. Since it invokes this.users.update it will need to become async (this.users.updateAsync ? not sure what the plan is here) and all invocations will need to adapt and also become async, which then ripples out via the call chain. What is the best way forward to convert javascript code to become async by analysing all call chains? Should there then be an intermediate sync version that just invokes Promise.await(destroyTokenAsync(...args)) and logs a warning? What is the execution plan here? |
Beta Was this translation helpful? Give feedback.
-
|
I think that the approach that @filipenevola has chosen for MongoDB in #11605, i.e., creating |
Beta Was this translation helpful? Give feedback.
-
|
Note that |
Beta Was this translation helpful? Give feedback.
-
|
Only the low-level API would be deprecated, so basically the hooks themselves. The main functionality of context tracking would still be exposed through |
Beta Was this translation helpful? Give feedback.
-
|
Generally speaking, a project like Meteor should only ever implement |
Beta Was this translation helpful? Give feedback.
-
|
Just ran into the issue with wasm today and found #11143 in the process of debugging. I might consider putting some upwork or internal development resources towards helping make progress. |
Beta Was this translation helpful? Give feedback.
-
|
Hey everyone, we finally have updates here! I just want to reassure we are continuing the work to change the way Meteor executes async code. This is a gradual work with many pieces and many people involved. Please read the initial plan and comments on this discussion to understand the context of this work. Plan PhasesPhase 1 - Node.js 14 [FINISHED]Phase 1.1 - Fibers running on Node.js 16 [CANCELED]Phase 2 - Make Fibers usage in Meteor server application runtime optional [CURRENT]Steps to complete this phase:
Expected Result
Phase 3 - Make Fibers usage in Meteor tools optional
Phase 4 - Declare Meteor packages using Fibers in maintenance mode
At the moment, the great people directly involved in this work are @filipenevola (Quave), who will lead the plan and bring another dev from Quave, @radekmie from Vazco, @denihs (and myself) from Meteor, and @zodern from Monti APM. We hope to make more progress in the mentioned steps in the next weeks. We really appreciate the recent feedback about this work from people like @StorytellerCZ, @menewman, @jankapunkt, @rochdev, @perbergland, @Julusian, and many others. I think if we centralize the discussion here and in the specific PRs that will help everyone to follow the progress. Please reach out to me if you see something where you could collaborate. If you think we are not considering something important, please comment here. This work is our main focus now, and it will be in the next couple of months. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I think we'll need to adopt some tools for working without Fibers, but in async mode. For example, Meteor uses Maybe we can add a new label like "Remove Fibers" and add a separate issue for such things like above? |
Beta Was this translation helpful? Give feedback.
-
You are right, we will probably have to re-implement This work is going to happen inside this Phase:
|
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! Some updates about the top level await support. To refresh our memory a bit: the top level await support is required to be able to wait for the To give more context, let's see a bit of how Meteor's build system works: before evaluating any module/code, we use Babel to check for syntax errors. This means that just swapping the current You can check how Meteor bundles the code by checking the So the steps planned to support TLA are:
For the 2nd point, the 2nd approach is the most promising one, as from what we tested it works. This means that the example we gave above would be changed to: This is our current plan. If you guys have any suggestions/recommendations, please let me know your thoughts. |
Beta Was this translation helpful? Give feedback.
-
|
@matheusccastroo +1 for no.2 |
Beta Was this translation helpful? Give feedback.
-
|
I currently am working on top level await. I've been focused on a different aspect: what limitations the legacy client add, and how to ensure a module's dependencies are loaded before the module is ran. With what I currently know, I think we can almost fully implement top level await in reify. I'm planning to start writing some code this week to check. |
Beta Was this translation helpful? Give feedback.
-
|
I'm putting you both in contact @zodern, so we can also join forces here! @matheusccastroo is from Quave and is helping us on the Fibers migration as well. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, quick bi-weekly update: Here are the latest updates on the successfully completed tasks:
Currently, we are actively working on the following tasks:
You can follow the updates on our Fibers Public Roadmap and on this Github PR |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, quick bi-weekly update: Here are the latest updates on the successfully completed tasks:
Currently, we are actively working on the following tasks:
You can follow the updates on our Fibers Public Roadmap and on this Github PR |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone! Here is a small update: Some tasks we've been working on it:
Our plan for the next week is to release one more Alpha, which will include changes like this one and this one. After that, we'll address the issue Unhandled Rejection when fails to bundle the app as well as more issues our community is reporting, like this one. See you soon for more updates! You can follow the updates on our Fibers Public Roadmap and on this Github PR . |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone! Here is another update for you! In the past few weeks, we addressed issues like this, and we have a new Alpha, We now have a new document where you can see the main tasks we need to finish until we have a final release for version 3.0. You can check that out here. Also, for the next two weeks, the main tasks we'll be addressing are Unhandled Rejection when fails to bundle the app and Review Meteor.call, Meteor.callAsync, Meteor.apply, Meteor.applyAsync so we can release a final Alpha version. See you soon for more updates! You can follow the updates on our overview doc, Fibers Public Roadmap and on this Github PR . |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm missing the bi-weekly updates. Are they moved somewhere else? I was also looking at the overview doc and the Fibers Public Roadmap but they haven't updated for multiple weeks now. To be more specific: We were planning to do the migration to 3.0 in Q1/2024. But since there is no beta yet we probably move it to Q2/2024. Do you have schedule for a beta release? |
Beta Was this translation helpful? Give feedback.
-
|
Here are some newer information: #12865 |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone! Here is a quick update for you on what we've been doing in the past few days! Our We've also been working with HUGE thanks to our community on this one. We had a lot of feedback and participation on it. And special thanks to @zodern, who created this code. You can check their post about it here. We also started working on our Migration guide and updating our Docs for version 3. Unfortunately, we had some drawbacks in the past couple of weeks, but we're still confident that we can have a Beta version by the end of the year. So stay tuned, and see you soon for more updates! You can follow the updates on our overview doc, Fibers Public Roadmap and on this Github PR . |
Beta Was this translation helpful? Give feedback.
-
|
Great work, @denihs. Is the API for callAsync/applyAsync considered stable now? A blog post dedicated to this would be excellent. I could help with that, but we need the confirmation that it's the final API first. |
Beta Was this translation helpful? Give feedback.
-
|
Now, in beta zero, we can say we have a stable version. I think would be good to have this blog post about it |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone! Here is a quick update on what has happened in the last month! We launched the first Beta version 🎉!
Meaning that we finished the work with our Something worth mentioning again is that this version is already running with Node 20! Also, we're working hard to finish migrating and updating our docs and guides. You'll hear more about it soon. We're now moving forward with the Beta, and your help with tests and feedback will be of great value! So stay tuned, and see you soon for more updates! You can follow the updates on our overview doc , Fibers Public Roadmap and on this Github PR |
Beta Was this translation helpful? Give feedback.
-
|
Are there any plans to continue to extend security support for Node 14 past this April? Will the eventual release date of Meteor 3 affect security support extensions? |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone! Here is another update on what we've been working on in the past few weeks. We fixed an issue where Symbol.asyncIterator on Cursor was not working. We updated the Reify package inside the core where it wasn't updated yet. Also worked on issues like: Insert not working for collection created of the fly. Right now, we are working on:
We also moved forward with the docs, and it's looking really good. Here is a sneak peek:
We are planning on releasing a Beta 1 pretty soon, so stay tuned! You can follow the updates on our overview doc , and on this Github PR. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone! Here is one more update. We release one more Beta, this one being beta.4. It includes:
We're already working on the tasks for the next one, so stay tuned! You can follow the updates on our overview doc , and on this Github PR. |
Beta Was this translation helpful? Give feedback.
-
|
Just went to give this a try. This is the first time I've tried updating to the new release. |
Beta Was this translation helpful? Give feedback.
-
|
@wreiske There is a forum discussion with some suggestions for it: https://forums.meteor.com/t/trouble-updating-to-3-0-alpha-17-minisat/60728 Curious if you’re able to pinpoint a solution. |
Beta Was this translation helpful? Give feedback.
-
|
@wreiske @jamauro I think the root of the problem is a technical one in the dependancy resolution system; there is some sort of constraint satisfaction solver that tries to work iteratively and drowns itself with memory. That being said, the way you get around this is by
The process is a little tedious, but I only needed to do it once. It is also a great way of finding redundant/ outdated/ unused dependencies in your project. |
Beta Was this translation helpful? Give feedback.
-
|
I can see in the PR that |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! Looking forward to the eventual RC 🫡 |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Is there a problem with the dependencies of mongo-decimal package. We need to change it to: Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Same problem with less package: The strange thing is that if I copy the less package locally without any modifications, it works with the wrong dependencies. It seems that all packages under packages/no-core are not handled during the distribution phase. |
Beta Was this translation helpful? Give feedback.
-
|
I've published a new |
Beta Was this translation helpful? Give feedback.
-
|
@denihs @Grubba27, in terms of migration, are we still expecting any of the latest async changes for 3.x branch to be backported to 2.x branch? |
Beta Was this translation helpful? Give feedback.
-
|
I've just started testing our applications on beta.6 and I've encountered an inconsistency regarding the documentation. Specifically, it's about the handling of the |
Beta Was this translation helpful? Give feedback.
-
|
yep, beta 7:
beta 6:
It is in the announcement for Beta 7 |
Beta Was this translation helpful? Give feedback.
-
|
Ok, missed beta.7 - sorry for disturbance ;) |
Beta Was this translation helpful? Give feedback.
-
|
I've noticed there's also beta.8 on npm - should I test with it or stick with "official" beta.7? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, that was an issue we had. The npm version is off by one(npx meteor@beta / npx [email protected] will download the beta.7)
That was a valid question; Beta 7 was only officially launched yesterday. |
Beta Was this translation helpful? Give feedback.
-
|
✍️ I just realized the matching between official Meteor 3.x beta version and NPM CLI. I had to bump the version of CLI twice to fix a last issue with CLI and this got unsync. Good to know this for next versions planning. |
Beta Was this translation helpful? Give feedback.
-
|
Updates about the first Meteor 3.0 RC Release: Due to some roadblocks, our ETA for the RC Release has been postponed to 19th April. These are the tasks for the first RC:
Estimated Release Dates
You can follow the updates on What's left until an official Meteor 3.0? Discussion, and on the Release 3.0 PR. |
Beta Was this translation helpful? Give feedback.
-
|
It is time for a Weekly update 🎉 ☄️ We are proud to say that we have the first Release Candidate for Meteor 3.0 🤩🥳 The priority of the next RC will be to review and fix issues with Cordova's experience. Next Releases:
You can follow the updates on What's left until an official Meteor 3.0? Discussion, and on the Release 3.0 PR. |
Beta Was this translation helpful? Give feedback.
-
|
It is time for a Weekly update 🎉 ☄️ We are proud to say that we have the Release for 2.16 beta 🤩🥳 Tasks and fixes for the next Meteor 3.0 RC:
Next Releases:
You can follow the updates on What's left until an official Meteor 3.0? Discussion, and on the Release 3.0 PR. |
Beta Was this translation helpful? Give feedback.
-
|
It is time for a Weekly update 🎉 ☄️ We are proud to say that we have the Release for Meteor 3.0-rc.1 🤩🥳 We also have the Meteor 2.16-rc.0 🤩🥳 Tasks and fixes for the next 3.0 RC:
Next Releases:
|
Beta Was this translation helpful? Give feedback.
and others
-
Plan to change how Meteor executes Async code
Related PRs: #11605
Goal
The goal of this discussion is to define a plan together with the community on how to proceed with
asyncexecution in Meteor.js.We, Meteor Software, are proposing a plan but we are open to ideas and improvements on this plan.
How the community can help?
Context
Fibers was the choice for Meteor as at the time there was no async/await standard available, you can understand more this choice by watching this talk (slides).
Meteor uses Fibers to implement async behavior when you use Fibers directly or even when you use async/await keywords or a Promise. This enabled Meteor to use new patterns very early.
In recent years some conflicts between Node.js and Fibers started to appear and then the need for a different approach.
Examples of conflicts:
Our desired approach is to follow what was adopted as the standard for async code in Node.js but this has a fundamental challenge as the new standard doesn't allow the usage of
awaitkeyword unless we are inside anasyncfunction.The challenge here is that in some cases this requirement is going to affect many calls layers in the code. This change will affect Meteor tools, Meteor core packages, and user application code as well.
The good news here is that most of the packages using Fibers don't expose its usage so it's not going to affect user's code.
So far we have not identified many cases where user application code is going to be affect and we also have plans to automatize and minimize the impact of this change, probably the common use case will be around MongoDB calls that would need to be awaited.
Next we propose a few phases to rollout these changes, the idea is to minimize as much as possible breaking changes and also to provide ways to give plenty of time for developers to adapt their code.
Plan Phases
First the goal is to make Meteor compatible with Node.js LTS for as long as possible using Fibers and later remove Fibers as a dependency but without the pressure of Meteor not working with Node.js LTS.
Phase 1 - Node.js 14 [FINISHED]
This phase was completed on Meteor 2.3
Expected Result
Meteor working with LTS Node.js using Fibers until April 2023. We do not expect to fix possible conflicts with async_hooks and WebAssembly at this point.
Phase 1.1 - Fibers running on Node.js 16 (optional)
We could extend for how long Meteor will be running with Fibers on Node.js LTS versions by supporting Node.js 16 as well, for that we have a few challenges that we need to address:
It's important to notice that this phase it not required, it could be executed in parallel with the Phase 2 or even skipped.
Expected Result
Meteor working using Fibers with Node.js 16. We do not expect to fix possible conflicts with async_hooks and WebAssembly at this point.
Phase 2 - Make Fibers usage in Meteor server application runtime optional
Meteor uses Fibers on server runtime and also in the Meteor tools. The goal of this phase is to make Meteor server application runtime possible without using any Fibers.
The plan is to not force all the applications to migrate to this new way of running async code altogether. We are planning in a way where applications could change a few packages each time.
Of course, the consequence of having a few packages still relying on Fibers is that while there is at least one package in an application relying on Fibers the entire application will need to run in the version of Meteor where Fibers is still accepted and compatible.
The good news here is that most of the packages using Fibers don't expose its usage so it's not going to affect user's code.
Here are the steps to complete this phase:
boot.jsuses a Fibers to start the server. We already have some early results removing thisboot.jsusage here. In this commit it's possible to generate the boilarplates and run the Meteor server without the global Fiber.Promiseimplementation of Meteor that uses Fibers toPromiseFiber.PromiseFiber.bindEnvironmentandwrapAsyncin external async calls.asyncandawaitare used in the application code.Packages using Fiberssection below for details.-asyncin the name providing the same functionality using standard async.mongo-asyncpackage would provide Meteor collections where is necessary to await the calls to MongoDB.-asyncpackage when creating new projects withmeteor create.Expected Result
Meteor apps will be compatible with
async_hooksandWebAssemblyand any other Node.js modules as Meteor apps are not going to be using any non-standard technology anymore in application code.This phase is not going to require a full migration in a single step. Applications can migrate group of packages in incremental steps, guides are going to be provided as soon as a package is ready to be replaced.
With this phase completed the applications should start to use the new
-asyncpackages and adoptasyncandawaitkeywords where it's necessary in the new packages.Also package authors from the community should be able to have many examples on how to migrate their packages if they are relying on Fibers directly. Application code that is using Fibers directly should also adapt to regular
asyncandawaitkeywords instead.At this point Meteor tools will still using Fibers but this is not going to affect application code in anyway but this would prevent Meteor to migrate to a newer version of Node.js that is not compatible with Fibers, until the next phase is completed.
Phase 3 - Make Fibers usage in Meteor tools optional
Meteor tools use Fibers to execute async code. We want to remove this dependency as well. It's important to understand that this dependency does not cause problems in application code but as Meteor tools are an important part of Meteor platform we also want to remove Fibers from it. Otherwise this could keep locking Meteor in an old version of Node.js because of Fibers' conflicts.
Here are the steps to complete this phase:
Tools using Fiberssection below for details.Expected Result
Meteor will be compatible with async_hooks and WebAssembly and any other Node.js modules not only in application code but also in Meteor tools.
Phase 4 - Declare Meteor packages using Fibers in maintenance mode
At this point Meteor is no longer dependent of Fibers so we would not need to keep adding features and updating core packages depending on Fibers as they already have a new version without Fibers.
This doesn't mean that we are not going to fix bugs in these packages but they are going to be considered
maintenance only. If new features are implemented they are going to be added only in the-asyncversion.Expected Result
Meteor continue to evolve without having to provide new features to two versions of the same package.
This also is going to provide stability for applications that are no longer receiving new features and probably don't need to have latest features.
We are open to feedback on this plan and we will be glad to answer any questions.
See below two sections listing usages of
Fiber,FutureandPromise.awaitin packages and later in Meteor tools.Packages using Fibers
wrapYieldingFiberMethods1Tools using Fibers
* maybe we can remove this usage instead of migrate** it's probably a very important usageLink to the previous Meteor Feature Request about this.
Beta Was this translation helpful? Give feedback.
All reactions