RCORE-2064 eq neq string compression #7575
Conversation
nicola-cab
force-pushed
the
nc/eq_neq_string_compression
branch
from
2c40d1f
to
a337502
Compare
There was a problem hiding this comment.
-
We need to also handle the situation where a string has been interned, but not all leaves have been compressed with the interned value.
-
I think there is something wrong with the insensitive search here. It should be able to handle mixtures of upper/lower case.
|
|
||
| size_t _find_first_local_interned_string(const StringData& data, size_t start, size_t end) const | ||
| { | ||
| REALM_ASSERT_DEBUG(m_leaf->is_interned()); |
There was a problem hiding this comment.
This will not hold.
Even though a string has been interned, we may have leafs with strings which are not interned (yet).
Consequently for every leaf a check has to be made and
A. if the leaf has been interned, we can compare integers
B. if the leaf has not been interned, we have to compare the strings, even though it is slower.
test/test_query.cpp
Outdated
| tv = q.find_all(); | ||
| CHECK_EQUAL(tv.size(), 1); | ||
|
|
||
| q = table->where().not_equal(col_str, StringData("TEST"), true); |
There was a problem hiding this comment.
When testing case insensitive search, we should test with a mix of upper/lower case. I think this will reveal a bug in the implementation in this PR.
finnschiermer
force-pushed
the
fsa/string-interning
branch
from
5500793
to
7ae8865
Compare
What, How & Why?
String interning introduces a new compressor for strings, which basically implements a function that maps the string to some unique integer.
For compressed strings and in particular for EQ and NEQ, we can leverage this in order to quickly compare integers against integers.
In all the other cases, we need to decompress the string.
TODO: