How to do grid wide processing per partition?

[javafanboy]

The documentation describes quite clearly how to run processors of different kinds for a specific key, for all keys or for each cache node but what about if I want to process the data in each partition separately?

Lacking any "out of the box" solution to do this my current idea is to use an AbstractInvocable that will run on each storage enabled node and then let it find the partitions of the desired cache(s) on that node and do whatever I want to do with them one by one or in parallel (for instance using virtual threads).

I have so far not found how to find the number of partitions on each node (not necessarily perfectly evenly balanced between nodes) or indeed the actual keys/values for each partition :-(

So far I have written this code for my Invocable

InvocationService service = (InvocationService) getService();
cluster = service.getCluster();
Service cacheService = cluster.getService("CacheService");
if (cacheService instanceof CacheService) {
    BackingMapManagerContext context = ((CacheService) cacheService).getBackingMapManager().getContext();
    // ??????
}

Is this the right way to access what I want and if so how do I get further? Cant really find the right methods from here?

[thegridman]

The question would be what you intend to do with the BackingMapManagerContext you have.

You can run entry processors (or aggregators) against specific partitions using a PartitionedFilter see https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.2206/java-reference/com/tangosol/util/filter/PartitionedFilter.html

[javafanboy]

Wow super fast responce "Gridman" - thanks!

Will investigate that direction further!

[thegridman]

Another approach to target a single partition is to create a class that implements PartitionAwareKey. You can then use that as the key to execute an entry processor. When you execute an EP, the entry does not have to actually exist, so this does not have to be the same as the actual cache keys. This can be useful to target an EP to a single partition, so you can create a PartitionAwareKey with a specific partition id, then from inside the EP, you know your partition and you can access whatever you want (entries, associated caches etc) from the EP.

[javafanboy]

That also sounds worth looking into. I actually have multiple use-cases on my "radar" so different ones may come in handy...

What is by the way the simplest way to find the configured number of partitions programmatically?

[thegridman]

To get the partition count programatically you need to be in a cluster member. Then you can get the service for the cache you are interested in, cast it to a PartitionedService and call getPartitionCount(). From a client you don't know the partition count directly, so you could to do it via a simple EntryProcessor call that gets the partition count and returns it. As this EP is only getting the partition count from the context in the entry, you can just invoke it against some fictitious non-existent key if you wanted.

[javafanboy]

Another approach to target a single partition is to create a class that implements PartitionAwareKey. You can then use that as the key to execute an entry processor. When you execute an EP, the entry does not have to actually exist, so this does not have to be the same as the actual cache keys. This can be useful to target an EP to a single partition, so you can create a PartitionAwareKey with a specific partition id, then from inside the EP, you know your partition and you can access whatever you want (entries, associated caches etc) from the EP.

I created a special cache that weill not contain any data and only will be used to be able to invoke processors using "dummy" partition aware keys (one for each configured partition) and using that I was able to invoke an entry processor once for each partition. In order to do something meaningful I do however need to get to the binary data of my real caches for each partition. Once again I am a bit lost in all the managers, contexts and other Coherence classes I can get to :-(

I assume I should turn the entry passed into the processor (invoked on the non-exisitng partition aware key) into a binary entry (hopefully this is possible also for a at this point non-exisitng entry!) but where to go from there?

[thegridman]

Which version of Coherence are you using, because some simpler ways to get related cache entries were added in newer versions.

[javafanboy]

I do these experiments with latest version.

…

On Wed, Apr 24, 2024, 18:04 Jonathan Knight ***@***.***> wrote: Which version of Coherence are you using, because some simpler ways to get related cache entries were added in newer versions. — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADXQF77GQNT4MEZZOOK7JDY67JYJAVCNFSM6AAAAABGWVAQMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGMYDEOJYGI> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[thegridman]

OK, here is a snippet for an entry processor that gets entries from another cache

public class MyProcessor<K, V, R>
        implements InvocableMap.EntryProcessor<K, V, R>
    {
    @Override
    public R process(InvocableMap.Entry<K, V> entry)
        {
        BinaryEntry<K, V> binaryEntry = entry.asBinaryEntry();
        String key = "foo";
        
        // Get the entry for update (the entry will now be locked)
        BinaryEntry<Binary, Object> otherEntry = binaryEntry.getAssociatedEntry("some-other-cache", key);
        
        // Get the entry for reading only (the entry will NOT be locked)
        BinaryEntry<Binary, Object> roEntry = binaryEntry.getAssociatedEntry("some-other-cache", key, true);
        
        return null;
        }

• You basically, convert the entry to a BinaryEntry with entry.asBinaryEntry() so you can then get the contexts, etc
• Then to get a related entry in the same partition in another cache call binaryEntry.getAssociatedEntry() which takes a cache name and the non-serialised key.
• Or if you know you only want to read that entry then get it read only so it is not locked.

One thing to be aware of if you intend to enlist multiple other cache entries and you might have multiple instances of this entry processor executing that might try to enlist the same entries. You need to avoid deadlocks - Coherence will throw an exception if two EPs try to enlist the same entries but in different orders to avoid a deadlock. But to really make sure, it is safest to sort the keys that you will enlist, so all EP instance lock keys in the same order, then they will just wait for each other. For read only entries, this does not matter as they are not locked.

[thegridman]

I updated the comment above because the new APIs to get a related entry binaryEntry.getAssociatedEntry() take the key as a non-serialised value (whereas the other APIs on the BackingMapContext take Binary keys).

[javafanboy]

Thanks a lot for the nice example - will build on it! Are there any article or other resources where one can learn more about this type of low level Coherence programming. The Java docs are a bit brief and in situations where you need to cast it is even harder to figure it out....

…

On Wed, Apr 24, 2024, 18:30 Jonathan Knight ***@***.***> wrote: I updated the comment above because the new APIs to get a related entry binaryEntry.getAssociatedEntry() take the key as a non-serialised value (whereas the other APIs on the BackingMapContext take Binary keys). — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADXQF5WJIHV65EQ7X4TJN3Y67MYXAVCNFSM6AAAAABGWVAQMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGM3TIMRZGE> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>