New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
collection bindings OOM for large data sets #214
Comments
The first approach is very expensive, because a cross product of two countries with all artists are produced, whereas the latter two options does a natural join of these. Natural join is a lot cheaper than cartesian product. A possible optimization is to automatically translate the first to the laters. This is going to be a more advanced optimization that is further down the line. The next release of the optimizer probably won't have this feature, as we are focusing on optimizing the where clauses and simple bindings. The optimization of collection bindings will wait after that is done. |
From clojurian datalevin channel:
andersmurphy 4:56 AM
So I’m finding when I use collection bindings. I.e pass a collection and :in $ [?x …] I’m much more likely to run out of memory, even if the collection contains a single value. Inlining the values and performing an or doesn’t result in running out of memory. Is there something that makes collection bindings inherently expensive?
This implementation does run out of memory (with large datasets):
The implementations bellow don’t run out of memory (with large datasets):
or
The text was updated successfully, but these errors were encountered: