RCORE-2094 Compressing Integer Arrays #7668

nicola-cab · 2024-05-01T17:51:31Z

What, How & Why?

Main Logic for compressing integers. it includes:

https://jira.mongodb.org/browse/RCORE-2051 ==> Packed Format.
https://jira.mongodb.org/browse/RCORE-2052 ==> Flex Format
https://jira.mongodb.org/browse/RCORE-2053 ==> Adding the integer compressor class for compressing in Packed or Flex
https://jira.mongodb.org/browse/RCORE-2054 ==> Main Logic for compressing and decompressing Arrays
https://jira.mongodb.org/browse/RCORE-2056 ==> Some query optimizations and faster decompression of Arrays

…rectory_current_core

…out-experiment

…/new-layout-experiment

…ec there :-(

…ut_local_directory

coveralls-official · 2024-05-01T18:33:30Z

Pull Request Test Coverage Report for Build nicola.cabiddu_1715

Details

1479 of 1721 (85.94%) changed or added relevant lines in 35 files are covered.
103 unchanged lines in 24 files lost coverage.
Overall coverage increased (+0.002%) to 90.611%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/realm/array_mixed.cpp	1	3	33.33%
src/realm/array_unsigned.cpp	11	13	84.62%
src/realm/integer_flex_compressor.cpp	31	40	77.5%
src/realm/integer_packed_compressor.cpp	18	27	66.67%
src/realm/query_engine.hpp	3	12	25.0%
src/realm/array.cpp	110	125	88.0%
src/realm/integer_packed_compressor.hpp	106	127	83.46%
src/realm/integer_compressor.hpp	78	106	73.58%
src/realm/array_aggregate_optimizations.cpp	141	176	80.11%
src/realm/integer_compressor.cpp	188	224	83.93%

Files with Coverage Reduction	New Missed Lines	%
src/realm/array.hpp	1	95.73%
src/realm/array_unsigned.cpp	1	85.5%
src/realm/sync/noinst/server/server_history.cpp	1	63.7%
src/realm/util/compression.cpp	1	89.79%
src/realm/util/file.cpp	1	78.84%
test/test_table.cpp	1	99.52%
src/realm/array_with_find.hpp	2	97.92%
src/realm/object-store/shared_realm.cpp	2	91.89%
src/realm/sync/network/http.hpp	2	82.27%
src/realm/sync/transform.cpp	2	60.99%

Totals
Change from base Build nicola.cabiddu_1713:	0.002%
Covered Lines:	216702
Relevant Lines:	239157

💛 - Coveralls

jedelbo

Some initial comments

jedelbo · 2024-05-23T11:38:26Z

src/realm/array.cpp

- }
- }
- start += sizeof(int64_t) * 8 / no0(w) * chunks;
+ if (!is_compressed() && m_integer_compressor.get_encoding() == NodeHeader::Encoding::WTypBits) {


I think the first condition is redundant.

yep, It is too defensive.. You are right…

jedelbo · 2024-05-23T11:42:29Z

src/realm/integer_compressor.cpp

+ for (size_t i = 0; i < sz; ++i) {
+ auto item = arr.get(i);
+ values.push_back(item);
+ REALM_ASSERT_DEBUG(values.back() == item);


Perhaps an assertion too much. Are you in doubt that push_back works?

Yep too conservative, originally I was in doubt whether our integer compression algo worked :-) , but there is also an assertion (only in debug mode) that checks the whole array after it gets committed, so I can remove this.

jedelbo · 2024-05-23T11:46:51Z

src/realm/integer_compressor.cpp

+ REALM_ASSERT_DEBUG(values[indices[i]] == arr.get(i));
+ }
+
+#if REALM_DEBUG


Is this not tested above?

jedelbo · 2024-05-23T11:47:49Z

src/realm/integer_compressor.cpp

+ auto last = std::unique(values.begin(), values.end());
+ values.erase(last, values.end());
+
+ for (size_t i = 0; i < arr.size(); ++i) {


jedelbo · 2024-05-23T13:11:49Z

src/realm/CMakeLists.txt

@@ -13,6 +13,7 @@ set(REALM_SOURCES
 array_blobs_big.cpp
 array_decimal128.cpp
 array_fixed_bytes.cpp
+ array_aggregate_optimizations.cpp


The files should be added to Package.swift file too.

jedelbo · 2024-05-23T13:13:55Z

src/realm/array.cpp

@@ -190,38 +189,108 @@ using namespace realm::util;

 void QueryStateBase::dyncast() {}

-size_t Array::bit_width(int64_t v)
+Array::Array(Allocator& allocator) noexcept


Why not inlined anymore?

jedelbo · 2024-05-24T09:36:06Z

src/realm/integer_flex_compressor.hpp

+ const auto mask = c.v_mask();
+ BfIterator ndx_iterator{data, offset, ndx_w, ndx_w, start};
+ BfIterator data_iterator{data, 0, v_w, v_w, static_cast<size_t>(*ndx_iterator)};
+ while (start < end) {


I think we should use the same approach as in the parallel find, where we first search the values and then search for the relevant indices afterwards.

mmm, interesting.. I am not sure we are going to become faster or slower.. I'll try..

jedelbo · 2024-05-24T14:00:36Z

src/realm/array.hpp

@@ -176,19 +176,21 @@ class Array : public Node, public ArrayParent {
 template <size_t w>
 void set(size_t ndx, int64_t value);

- int64_t get(size_t ndx) const noexcept;
+ inline int64_t get(size_t ndx) const noexcept;


Does it make sene to add inline to a function definition? It happens a lot in this PR.

no, because the getter is defined below, it is a left over, I'll delete it ..

jedelbo

I am missing more basic testing of the compressed arrays, Group_ArrayCompression_Correctness is hardly enough.

nicola-cab and others added 30 commits December 21, 2023 12:24

Merge branch 'master' of github.com:realm/realm-core into nc/local_di…

03555ef

…rectory_current_core

silence some warnings

38d8f79

Merge branch 'master' of github.com:realm/realm-core into nc/local_di…

d337695

…rectory_current_core

comment check for failing test

5163909

Merge branch 'master' of github.com:realm/realm-core into fsa/new-lay…

2c4e0c4

…out-experiment

Merge branch 'master' of github.com:realm/realm-core into fsa/new-layout

59cf041

Merge branch 'fsa/new-layout' of github.com:realm/realm-core into fsa…

bc5e348

…/new-layout-experiment

merge new-layout-experiment

10549e1

min code cleanup

07174c0

point fix find for encoded arrays

c8207b0

copy values directly instread of calling recursively arr.set()

a9bfef7

avoid to sign the restored value

357c08f

start of classifying arrays for compression

00ddb0f

classification down to column types

eca9697

fix find for compressed arrays

e3d9a73

reworked to rely on passing Table down to leafs, cause we need the Sp…

b810028

…ec there :-(

revert changes for unsigned arrays that broke links

4a3b83a

avoid COW while decompressing

f627f49

silence some warnings

bdc9b33

Merge branch 'master' of github.com:realm/realm-core into nc/new_layo…

195c0cf

…ut_local_directory

silence more warnings

1714c40

first attempt to cut through the BPlusTree madness

fb058f0

fix decompression width computation

4444f57

return unsigned every time the array is in compressed format

c9a3b14

Merge branch 'master' of github.com:realm/realm-core into nc/new_layo…

5260988

…ut_local_directory

silence warnings

48a0396

[wip] start on 'type driven' write process

ab7249f

cleanup array flex

a8ff4b0

some optimizations for array flex

d70ef63

Merge branch 'master' of github.com:realm/realm-core into nc/new_layo…

3ae49df

…ut_local_directory

github-actions bot assigned nicola-cab May 1, 2024

nicola-cab added 15 commits May 13, 2024 15:15

avoid undefined behaviour for ndx and value masks

50d6184

Merge branch 'next-major' of github.com:realm/realm-core into nc/RCOR…

8103c39

…E-2094

Merge branch 'next-major' of github.com:realm/realm-core into nc/RCOR…

d700e89

…E-2058

Merge branch 'next-major' of github.com:realm/realm-core into nc/rcor…

c98fbca

…e-2057

Merge branch 'nc/RCORE-2058' into nc/rcore-2057

4319f89

attempt to fix windows builders

75ce5d0

Merge branch 'nc/RCORE-2058' into nc/rcore-2057

615921b

Merge branch 'nc/RCORE-2058' into nc/RCORE-2094

715ef7f

silence warning windows

511472f

silence warnings

78b09e2

Merge branch 'nc/RCORE-2058' into nc/rcore-2057

4b93e1b

Merge branch 'nc/RCORE-2058' into nc/RCORE-2094

e276c3f

fix compilation issues after merge next-major

3604ccb

Merge branch 'nc/rcore-2057' into nc/RCORE-2094

f5b7d78

change types after merge next-major

b6e7f3e

Base automatically changed from nc/rcore-2057 to next-major May 21, 2024 14:55

Merge branch 'next-major' of github.com:realm/realm-core into nc/RCOR…

6634970

…E-2094

nicola-cab force-pushed the nc/RCORE-2094 branch 2 times, most recently from 4004852 to 2b9969d Compare May 21, 2024 16:02

fix compilation problem after merge

4815f2e

nicola-cab force-pushed the nc/RCORE-2094 branch from 2b9969d to 4815f2e Compare May 21, 2024 16:34

nicola-cab added the no-changelog label May 21, 2024

Merge branch 'next-major' of github.com:realm/realm-core into nc/RCOR…

bfba325

…E-2094

jedelbo reviewed May 24, 2024

View reviewed changes

jedelbo requested changes May 27, 2024

View reviewed changes

nicola-cab and others added 3 commits May 27, 2024 14:03

code review

e05557e

Use only one inderection when accessing compressed arrays. (#7736)

227c1e6

fix support write compressed arrays in table for Group::write().

2864568

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RCORE-2094 Compressing Integer Arrays #7668

RCORE-2094 Compressing Integer Arrays #7668

nicola-cab commented May 1, 2024 •

edited

coveralls-official bot commented May 1, 2024 •

edited

jedelbo left a comment

jedelbo May 23, 2024

nicola-cab May 27, 2024

jedelbo May 23, 2024

nicola-cab May 27, 2024

jedelbo May 23, 2024

jedelbo May 23, 2024

jedelbo May 23, 2024

jedelbo May 23, 2024

jedelbo May 24, 2024

nicola-cab May 27, 2024

jedelbo May 24, 2024

nicola-cab May 27, 2024

jedelbo left a comment

RCORE-2094 Compressing Integer Arrays #7668

Are you sure you want to change the base?

RCORE-2094 Compressing Integer Arrays #7668

Conversation

nicola-cab commented May 1, 2024 • edited

What, How & Why?

coveralls-official bot commented May 1, 2024 • edited

Pull Request Test Coverage Report for Build nicola.cabiddu_1715

Details

💛 - Coveralls

jedelbo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jedelbo left a comment

Choose a reason for hiding this comment

nicola-cab commented May 1, 2024 •

edited

coveralls-official bot commented May 1, 2024 •

edited