MongoDB Transactions

[msavin]

Hey everyone,

It's still hard for me to believe that Meteor doesn't support MongoDB transactions. It was a really big feature when launched, and can potentially broaden the appeal of Meteor.

Looking from their API, it's a pretty simple set of steps to use:

1. start the transaction
2. list all the mongo operations you wish to perform
3. submit the transaction and wait for the response

Good reference with examples of how to use it: https://forums.meteor.com/t/solved-transactions-with-mongodb-meteor-methods/48677

For a Meteor implementation, we need to have the Fibers support working and maybe add a bit of sugar to it? At minimum, we shouldn't have to rely on internal APIs to have it working.

A few ideas to start the conversation...

Meteor.methods({
	"example1_purchaseItem": function (purchaseData) {
		const { invoiceId } = Mongo.Transaction(() => {
			const invoiceId = Invoice.insert({...purchaseData});
			const changeQuantity = Items.update(purchaseData.itemId, { ... })
			return { invoiceId, changeQuantity }
		}, errorCallback)

		return invoiceId;
	},
	"example2_purchaseItem": function (purchaseData) {
		// Everything in between transaction start and resolve shou
		Mongo.transaction.start("transaction1");

		const invoiceId = Invoice.insert({...purchaseData});
		const changeQuantity = Items.update(purchaseData.itemId, { ... })

		Mongo.transaction.resolve("transaction1");

		// this is not a part of the transaction
		Meteor.users.update({ ... })
		
		return invoiceId;
	}
})

I'm personally a fan of the example1, as anyone can pick up its use just seeing from a code sample, as well as easily migrate their existing code to support it.

---

I also have a specific use case with my jobs queue package, Steve Jobs, where I think this can really shine. The job state is stored in MongoDB, and the idea is, to perform all the DB operations that the job requires, and update the job state document, in one shot. For example:

• Stripe sends a webhook message that a charge was successful
• Meteor server registers a job to process this information
• In one DB operation, the job can then:
  • update the payment status
  • mark the order as ready for delivery
  • update the job document to say it was resolved successfully

[make-github-pseudonymous-again]

Using the async/await solution suggested in the forums, one can get:

Meteor.methods({
  'some.method.name'(itemId: string, smurfId: string) {
    return executeTransaction(async () => {
      const smurf = await Smurfs.rawCollection().findOne(
        {_id: smurfId},
        {session},
      );
      if (!smurf) {
        throw new Meteor.Error('not-found');
      }

      const meta = await Items.rawCollection().updateOne(
        {_id: itemId},
        {
          $pull: {'meta.y': smurfId},
        },
        {session},
      );

      ...

    });
  },
});

Reactivity still works, although maybe lag is larger, Meteor is able to pickup changes from Mongo's oplog (correct?). One can even throw in specific transaction options or session options if needed. What's annoying is that one has to pass the session around (forgetting one will just make the thing not atomic silently?) and specifying rawCollection() all the time. One could make wrapper objects for Meteor collections so that ALL database interactions (insert, update, find, ...) magically forward it to rawCollection with the session but I did not go all the way there yet. Not sure what's a not too wasteful way to do it: we need to wrap ALL database interaction for ALL involved collections but most of my transactions will only use two or three verbs... Maybe a lazy getter. I do not know.

PS: I personally prefer try/catch and surf the async vibe:

"example3_purchaseItem": async function (purchaseData) {
  try {
    const { invoiceId } = await Mongo.Transaction(async () => {
      const invoiceId = await Invoice.insert({...purchaseData});
      const changeQuantity = await Items.update(purchaseData.itemId, { ... })
      return { invoiceId, changeQuantity }
    });
    return invoiceId;
  } catch (error: unknown) {
    ...
  }
},

[Torgen]

Anytime you update a collection outside of meteor's wrapper inside a method, call Meteor.refresh({collection, id}). It will make the method wait for the update to that document to be delivered to the client before returning.

[davidsavoie1]

@Torgen, where could I get some more information about Meteor.refresh? It seems undocumented...

Moreover, it seems I would need to provide a document id. If I use a transaction to update many documents at once (with updateMany for example, or multiple documents in different collections), would I then have to retrieve each document id and call Meteor.refresh for each?

Thanks for your feedback!

[Torgen]

It occurred to me that this might refer to a problem that is different from the one you have. The mechanism the subscription/method infrastructure uses to ensure that the effects of a method are published to the caller of that method on any relevant subscriptions before the method returns is Meteor.refresh. This is most relevant if your method has simulation, because without it the simulation will change the client side data, then the method will be called, the results will be returned without the side effects being published, the simulated changes will be rolled back, then later the effects will be received by oplog/polling and will become visible on the client side again. If you're using rawCollection and not separately implementing a client-side version, then the simulation roll back/roll forward flicker won't happen. You may still want to do it in that case if it's useful for the subscriptions to be updated before the method returns, but that doesn't guarantee a latency improvement.

Despite not having an underscore name, it seems like Meteor.refresh is very much internal and undocumented; from what I can tell, it takes a dict which is the selectors for updated documents, except that _id needs to be changed to id and you need a collection entry matching the collection name. It seems like you would need to collect all your selectors from your transaction and refresh them at the end. This is another reason why built-in transaction support would be better than a patched-on solution like the mongo-transactions package, since the collection methods could do that for you and refresh only after the transaction is committed.

[make-github-pseudonymous-again]

@Torgen This is most relevant if your method has simulation, because without it the simulation will change the client side data, then the method will be called, the results will be returned without the side effects being published, the simulated changes will be rolled back, then later the effects will be received by oplog/polling and will become visible on the client side again.

This is clear, thanks a lot for the explanation!

@Torgen If you're using rawCollection and not separately implementing a client-side version, then the simulation roll back/roll forward flicker won't happen.

Here I guess you mean not executing anything on the client, so that no changes are simulated.

[make-github-pseudonymous-again]

@Torgen Despite not having an underscore name, it seems like Meteor.refresh is very much internal and undocumented; from what I can tell, it takes a dict which is the selectors for updated documents, except that _id needs to be changed to id and you need a collection entry matching the collection name. It seems like you would need to collect all your selectors from your transaction and refresh them at the end.

This should be doable. I thought one had to list all the affected ids which would be hard. I see how this handles published data, but how does this non-published data that needs to be published due to changes (think pagination, newly created data, etc.).

@Torgen This is another reason why built-in transaction support would be better than a patched-on solution like the mongo-transactions package, since the collection methods could do that for you and refresh only after the transaction is committed.

Agreed.

[diavrank]

Also consider to add transactions support for Accounts methods like createUser, sendEnrollmentEmail, sendVerificationEmail, etc.

[diavrank]

and if possible, add transactions for Astronomy package

[StorytellerCZ]

Astronomy package is a community package, for anything related to it you will need to reach out to Astronomy maintainers: https://github.com/jagi/meteor-astronomy/

[diavrank]

It seems to be that Lukasz no longer want to add new updates like this. So, I was wondering if Meteor Community can fork his repository to add this feature. More info here.

[StorytellerCZ]

For that please start an issue here: https://github.com/Meteor-Community-Packages/organization/issues/new/choose

[radekmie]

I just want to point out one thing: handling of transaction failures (either transient or "actual" errors) is an important part of the business logic (you not always want to retry and sometimes you know it's safe to retry until success). "Hiding" it in the Mongo.Transaction or any other API would require an option to configure that.

That's why I'd rather have the access to the underlying API with an explicit creation of the session and passing it around. Whether or not we'll use Fibers or any other implicit storage (Fibers shouldn't be used since #11505) is an implementation detail and can be handled in the user space (i.e., through a community maintained package).

Also, since #11605 is already in progress, that may be much easier to expose this functionality. The only problem is to make sure it'll play well with the existing code, like server-generated IDs and schema validation (collection packages in general).

[make-github-pseudonymous-again]

Do I understand correctly that the goal of a Meteor wrap around MongoDB transactions would be to allow transactions to be simulated on the client without writing twice the amount of code (isomorphism)? In that case, it seems natural that not every detail of MongoDB's implementation of transactions would transpire.

[radekmie]

It's possible to do the same without extra magic. If the Mongo object would provide an API to create the transaction (on the server it'd use the real one, on the client a mock or even a placeholder) and accept it in "correct places", it'd be enough. Also, the client-side version could actually verify, that you're using a session object created with this API (e.g., by using a tagged object).

On the other hand, I'm not really sure whether I like this idea. It may suggest, that the transactions will happen also on the client-side, and that's far from true.

[red-meadow]

One thing related to transaction failures is handling operations callbacks (collection.insertOne(doc, callback)). It seems reasonable to me that inside a transaction all callbacks should be called with either result (if the transaction was successful) or error argument. So they should be deferred until the transaction is ended. But last time I checked (2020), MongoDB Node.JS Driver was implementing just the opposite: callbacks were fired immediately, regardless of the transaction success. So it was easy to get false positive calls before a failed operation.
I know, the consistency is important, but maybe Meteor could do better in this regard?

[radekmie]

I know, the consistency is important, but maybe Meteor could do better in this regard?

Well, I'd rather have it compliant with MongoDB instead of doing anything "better".

[bhunjadi]

Yeah, I'm not sure I get the whole point of deferred callbacks here and why would this be better in any way. I certainly wouldn't want the system to behave like that.

I wouldn't call these callbacks "false positive" because on atomic level they were legitimate; doc was inserted, albeit in the transaction that would eventually fail.

[msavin]

Maybe this discussion needs to be expanded into modernizing Meteor <-> MongoDB driver integration. I know there could be some breaking changes once Fibers are dropped. Maybe this can all make for a proper Meteor 3.0 release

[bhunjadi]

I wrote this package for our app and it is working as expected for more than a year now. I also mentioned this in the thread you linked.

It's basically the same as you proposed in your first example with Mongo.Transaction.

Example:

function insertInvoice(invoice, lineItems) {
    runInTransaction(() => {
        Invoice.insert(invoice);
        LineItems.insert(lineItems);

        const payment = createPaymentDoc(); 
        Payments.insert(payment);
    });
}

[msavin]

@bhunjadi I believe regular MongoDB operations in Meteor support manual callbacks for errors - so for example:

Invoices.insert({
	...data
}, (e,r) => {
	// callback
}

It would be nice if a callback can be passed in similarly, and then to have some controls to retry the transaction or just ensure it is aborted, etc.

[bhunjadi]

This is what I managed to solve in the PR, looks at the tests here

They were failing previously since runInTransaction did not know how to wait for these callbacks.

[Torgen]

It looks like you haven't published your package, and someone else who published from a fork isn't up to date. Are you planning to publish, or should I git submodule it into my packages directory?

[bhunjadi]

Yeah, I'm using it as a submodule right now. No plan to publish it for now.

[Torgen]

I forked your repo and added trivial client-side stubs so I could use transactions on the server side without losing simulation on the client side or having to duplicate the method implementation. I don't think they're generally useful, since there are corner cases I never run into, but that are possible in general. For example, if halfway through the transaction you discover that some invariant is violated, you can throw an exception. In simulation, the changes won't be rolled back immediately, but will be discarded when the server-side method call finishes. However, there's an option to Meteor.apply to abort the call if the simulation throws an error; in that case the partially applied simulation will be kept around even though the server call will never be made. I think this leaks memory since it's keeping track of a method call that will never be made, but since the functionality exists in the framework, it needs to be supported. This is also an argument for Meteor to provide an interface to transactions natively: It already knows how to roll back a change on the client, since it can roll back the simulated method results, so the framework can reuse that behavior for transactions in a way that a package can't.

[jkasevits]

I might be misunderstanding something here, but the question I have is why not make use of the official node mongo driver (rawCollection) to handle a case like this?

The fact that Meteor duplicates some of the official driver's API provides some benefits, but also comes at a cost. The benefits being that it works with minimongo, so enables optimistic updates, and operations are synchronous via Fibers, like most Meteor code. But the cost is rather heavy. Meteor needs to play catch-up with the offical node mongo driver and it will never be as feature complete as the official driver.

Mongo transactions have been available for quite some time already for Meteor users using rawCollection. Yes, it requires the use of async code. But it is already clear that Meteor will need to get rid of Fibers soon. So it seems to me that it does not make sense to add any new Fibers based code anymore. And if async will be required anyway, using rawCollection for transactions seems like an easy step.

As a more general note, I'd like to see Meteor promote the use of rawCollection a bit more, for example in the docs. Relying on the offical driver should be a very stable and tested way to interact with the db in Meteor, as Meteor apps are just node app at the end of the day. I don't think it should be regarded as using some obscure internal API that might break at any time.

[make-github-pseudonymous-again]

OK complete brain-fart on my side. I humbly ask for forgiveness.

You got to have a way to refer to the-mongodb-driver-collection-object-that-corresponds-to-the-meteor-collection-object though. Although you can probably cache the mapping instead of littering your codebase with .rawCollection() method calls?

[davidsavoie1]

I succesfully used Coll.rawCollection() and session.withTransaction from the NodeJS driver. Everything seemed to work very well... until using it with a remote MongoDB database.

With a local instance of MongoDB, the transaction writes to the database correctly and the changes are caught fast by the publications. However, with a remote database, a transaction could take up to 7 seconds before being reflected in the UI. It seems to the user that nothing is happpening at all for this entire duration.

An answer from @znewsham explains very well why this is happening. When using the Meteor collection methods, the changes are propagated fast to the connected clients, bypassing the database oplog notification. When using the native collection, Meteor doesn't know about the changes being made and will only react to the oplog.

This is fine on a single member replica set (local MongoDB), but will kill reactivity on a remote, production grade multi-cluster database.

So... The Meteor "magic" seems to come from the wrapping of collections. Please correct me if I understand it wrong.

I would be most grateful to get a simple solution to this problem.

[jamauro]

@davidsavoie1 would you mind embedding a screenshot of the znewsham answer you're referring to? I can't access that Microsoft teams link. Thanks!

[Torgen]

@davidsavoie1 See my answer in another thread in this discussion: #11737 (reply in thread)

[davidsavoie1]

Oups! Sorry @jamauro, I hadn't notice the erroneous link. I fixed it...

[a4xrbj1]

Are there any updates on this feature request or more clarity in terms of fiber and MongoDb?

[jamauro]

I just published a new package jam:mongo-transactions that was inspired by this discussion and builds on some of the ideas shared here. Here are some of the benefits:

• Write with the standard Meteor collection methods you're accustomed to. You don't need to worry about using rawCollection().
• Because it's a low-level solution, ID generation works as expected.
• You don't need to worry about passing session around. This package takes care of that for you.
• Works out-of-the-box with other packages that automatically validate on DB writes, like jam:easy-schema and aldeed:collection2.
• One simple API to use. Mongo has made things complicated with two APIs for Transactions, the Callback API and the Core API. This package defaults to using the Callback API as recommended by Mongo, but allows you to use the Core API by passing in autoRetry: false.
• Can be used isomorphically.
• Compatible with Meteor 2.8.1 and up, including support for Meteor 3.0+

Here's a quick example. As said above, you can surf the async vibe:

import { Mongo } from 'meteor/mongo';

async function purchase(purchaseData) {
  try {
    const { invoiceId } = await Mongo.withTransaction(async () => {
      const invoiceId = await Invoices.insertAsync(purchaseData);
      const changeQuantity = await Items.updateAsync(purchaseData.itemId, { $set: {...} });
      return { invoiceId, changeQuantity } // you can return whatever you'd like
    });
    return invoiceId;
  } catch (error) {
    // something went wrong with the transaction and it could not be automatically retried
    // handle the error as you see fit
  }
}

If you end up taking it for a spin or have ideas on how to make it better, let me know!

[jamauro]

@Torgen No, I didn’t see any issues there in my testing. The subscription will pick up the changes to the publication after the transaction commits.

[make-github-pseudonymous-again]

@jamauro This implementation is using Meteor.EnvironmentVariable to keep track of the session object. Isn't this going to fail for concurrent async calls (no fibers)?

[jamauro]

@make-github-pseudonymous-again the attempts to test concurrency are here and there's another below it. They are passing on Meteor 3.0-beta.0 and I can see that a unique session object is being created for each transaction and carried through to the operations within it. Let me know if you were thinking of another way to test it.

[jamauro]

This should be in core 😉

Agree, ultimately I'd like to see this in Meteor core. Would be great to hear what the Meteor core team thinks. @fredmaiaarantes @denihs @Grubba27

More generally, I think we shouldn't need to reach for rawCollection. I posted a discussion a while back to that end. Hopefully, after 3.0 is out, these gaps can be addressed.

[make-github-pseudonymous-again]

@jamauro Does it work because it uses two different connections then? I guess this is the only way to achieve parallelism anyway. I must admit I am not really sure I understand what these "environment variables" are tied to in fiber-less Meteor.