Support trie implemention for FieldIndex #18673
Conversation
| */ | ||
| public NonUniqueFieldIndexInTrie(IndexDefinition<T, V> indexDefinition) { | ||
| // Wrap the PatriciaTrie with Collections.synchronizedMap to make it thread-safe | ||
| mIndexTrie = new PatriciaTrie<>(); |
There was a problem hiding this comment.
What's the default parameters if you use an empty parameter list to create PatriciaTrie?
| synchronized (mIndexTrie) { // Synchronize on the trie map to ensure thread safety | ||
| mIndexTrie.compute(fieldValueStr, (key, set) -> { | ||
| if (set == null) { | ||
| set = Collections.newSetFromMap(new ConcurrentHashMap<>()); |
There was a problem hiding this comment.
Why not directly using a ConcurrentHashSet?
|
Automated checks report:
Some checks failed. Please fix the reported issues and reply |
| * @param <T> the type of object | ||
| */ | ||
| @ThreadSafe | ||
| public class IndexedSetTrie<T> extends AbstractSet<T> { |
There was a problem hiding this comment.
If the only methods you modified is the ctor and the getByPrefix, why not regarding IndexedSetTrie as a descedant of IndexedSet? Then you just need to change ctor and gerByPrefix, instead of copying all the methods.
| @@ -1039,15 +1038,11 @@ public void searchPagesByPrefix() throws Exception { | |||
| } | |||
|
|
|||
| private void checkPrefixSearch(UfsUrl rootUrl, UfsUrl prefixUrl) { | |||
There was a problem hiding this comment.
You should remove the @Ignore annotation of searchPagesByPrefix(), so the test will be run on github CICD.
thu-david
changed the title
|
Automated checks report:
All checks passed! |
thu-david
force-pushed
the
feature/WorkerMemoryOptimization
branch
from
76435ee to
56d54fd
Compare
…/thu-david/alluxio into feature/WorkerMemoryOptimization
|
I have fixed the conflict. |
|
alluxio-bot, merge this please. |
|
merge failed: |
|
alluxio-bot, merge this please. |
alluxio-bot
merged commit 4aeca16
into
Alluxio:feature/WorkerMemoryOptimization
What changes are proposed in this pull request?
I have added new implemention for fieldIndex when use pageid as key.
Why are the changes needed?
Because the hashmap use much memory when save so many same prefix id.
Does this PR introduce any user facing changes?
No.