Parallelize Soroban #4715

sisuresh · 2025-05-07T22:56:15Z

Description

Marked as draft because logic to refund after all transactions have been applied is missing. Additional testing and overall safety analysis is also required. There are also some TODOs in the code.

Checklist

Reviewed the contributing document
Rebased on top of master (no merge commits)
Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
Compiles
Ran all tests
If change impacts performance, include supporting evidence per the performance document

src/testdata/ledger-close-meta-enable-classic-events-v1-protocol-23-soroban.json

Copilot

Pull Request Overview

This PR introduces parallel processing for Soroban transactions by adding new preloading and parallel apply functions. Key changes include:

New methods in TransactionFrame, FeeBumpTransactionFrame, and various OperationFrame derivatives to support parallel execution.
Enhanced exception handling and diagnostic messaging in the parallel apply flows.
Updates to ledger and metrics management to support the new parallel execution patterns.

Reviewed Changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/transactions/TransactionFrame.{h,cpp}	Added functions for preloading and parallel applying Soroban ops.
src/transactions/RestoreFootprintOpFrame.{h,cpp}	Extended RESTORE footprint phase with parallel apply support.
src/transactions/OperationFrame.{h,cpp}	Introduced preload and parallel apply helper methods for operations.
src/transactions/InvokeHostFunctionOpFrame.h	Added parallel apply overrides for invoking host functions.
src/transactions/FeeBumpTransactionFrame.{h,cpp}	Updated fee bump path with parallel apply and enhanced error handling.
src/transactions/ExtendFootprintTTLOpFrame.{h,cpp}	Extended TTL update operations with parallel apply support.
src/testdata/ledger-close-meta*	Updated test data to adjust lastModifiedLedgerSeq values.
src/main/AppConnector.h	Fixed a typographical error in the comments.
src/ledger/SorobanMetrics.{h,cpp}	Updated metrics reporting using atomic loads.
src/ledger/LedgerTxn{Impl,}.{h,cpp}	Added new methods for handling restored keys from the hot archive.
src/ledger/LedgerManagerImpl.h	Added helper functions to support parallel apply stages.

src/transactions/FeeBumpTransactionFrame.cpp

src/ledger/LedgerTxn.cpp

src/ledger/LedgerManagerImpl.cpp

marta-lokhova · 2025-05-08T20:57:42Z

src/ledger/LedgerManagerImpl.h

@@ -175,6 +175,27 @@ class LedgerManagerImpl : public LedgerManager
        AbstractLedgerTxn& ltx,
        std::unique_ptr<LedgerCloseMetaFrame> const& ledgerCloseMeta);

+    UnorderedSet<LedgerKey> getReadWriteKeysForStage(ApplyStage const& stage);


Do we need randomized hashing in the output?

Not sure what you mean. Are you asking if this could be a set instead?

The output is only used for lookups.

sure, what I'm trying to confirm is whether the use of rand hasher is intentional here (as opposed to using std::unordered_set). UnorderedSet is a custom data structure we usually use to hash out ordering-related bugs in standard sets/maps (per this issue), but I'm not sure this applies here.

The randomization is also a DoS-prevention measure, to prevent people from being able to attack a hashmap with colliding inputs. The tree-based maps don't have this failure mode (but are slower overall).

Ah no this was not intentional. I think unordered_set is fine to use here, and the DoS angle Graydon mentioned isn't something we need to worry about here.

marta-lokhova · 2025-05-08T23:40:19Z

src/ledger/LedgerManagerImpl.cpp

+    std::vector<std::thread> threads;
+    for (size_t i = 0; i < stage.size(); ++i)
+    {
+        auto& entryMapByThread = entryMapsByThread.at(i);


main thread can invalidate this reference while other threads perform application, so it would be safer to move ownership of the map object to another thread and then back to main. I'd also recommend hardening the code by adding an invariant that only a specific thread id can access the map since the map is not synchronized in any way, so you want to make sure threads aren't actually sharing data.

Good point. Updated to use a unique_ptr.

marta-lokhova · 2025-05-09T00:06:50Z

src/ledger/LedgerManagerImpl.cpp

+{
+    UnorderedSet<LedgerKey> res;
+
+    for (auto const& txs : stage)


terminology is getting tricky! phase, stage, bundle are all starting to sound a bit similar. I wonder if there’s a way to abstract it away via an iterator of sorts if all you care about is iterating over all transactions, for example. We can pass a lambda to the iterator. This way we can try to minimize places in core that need to be aware of the internal structure of a parallel tx set.

just for context, I noticed there are lots of nested loops in the code making it harder to review, so I wonder if we can create an abstraction for tx sets to avoid bleeding internal details into other subsystems of core.

I added an iterator to ApplyStage. This gets rid of all but one of the nested loops. I didn't add something similar for the entire phase, but I don't think that's necessary here. Let me know what you think.

sisuresh · 2025-05-09T19:08:30Z

src/transactions/TransactionFrameBase.h

+    std::unique_ptr<TxEffects> mEffects;
+};
+
+typedef std::vector<TxBundle> Thread;


Most of the new types in this file should be moved to a new file.

src/transactions/TransactionFrameBase.h

src/ledger/LedgerManagerImpl.cpp

sisuresh · 2025-05-09T23:05:33Z

src/transactions/TransactionFrameBase.h

@@ -38,6 +40,222 @@ using TransactionFrameBasePtr = std::shared_ptr<TransactionFrameBase const>;
 using TransactionFrameBaseConstPtr =
    std::shared_ptr<TransactionFrameBase const>;

+using ModifiedEntryMap = UnorderedMap<LedgerKey, std::optional<LedgerEntry>>;
+
+struct ThreadEntry


Thread -> Cluster

Could you expand the comment on what this struct is? The name - ThreadEntry - is a bit generic so it's hard to tell how it's supposed to be used.

marta-lokhova · 2025-05-12T23:15:16Z

src/ledger/LedgerManagerImpl.cpp

+UnorderedSet<LedgerKey>
+LedgerManagerImpl::getReadWriteKeysForStage(ApplyStage const& stage)
+{
+    UnorderedSet<LedgerKey> res;


std::unordered_set

src/ledger/LedgerManagerImpl.cpp

marta-lokhova · 2025-05-12T23:17:09Z

src/ledger/LedgerManagerImpl.cpp

+LedgerManagerImpl::collectEntries(AppConnector& app, AbstractLedgerTxn& ltx,
+                                  Cluster const& cluster)
+{
+    std::unique_ptr<ThreadEntryMap> entryMap =


nit: auto entryMap

marta-lokhova · 2025-05-12T23:23:16Z

src/util/numeric.cpp

@@ -350,4 +350,5 @@ doubleToClampedUint32(double d)
    }
    return static_cast<uint32_t>(std::clamp<double>(d, 0, maxUint32));
 }
+


nit: accident?

marta-lokhova · 2025-05-12T23:26:12Z

src/transactions/TransactionFrameBase.h

@@ -38,6 +40,222 @@ using TransactionFrameBasePtr = std::shared_ptr<TransactionFrameBase const>;
 using TransactionFrameBaseConstPtr =
    std::shared_ptr<TransactionFrameBase const>;

+using ModifiedEntryMap = UnorderedMap<LedgerKey, std::optional<LedgerEntry>>;
+
+struct ThreadEntry


Could you expand the comment on what this struct is? The name - ThreadEntry - is a bit generic so it's hard to tell how it's supposed to be used.

marta-lokhova · 2025-05-12T23:42:31Z

src/transactions/TransactionFrame.cpp

+    DiagnosticEventBuffer& buffer) const
+{
+    releaseAssert(isSoroban());
+    releaseAssert(mOperations.size() == 1);


is this right? In

stellar-core/src/transactions/TransactionFrame.cpp

Line 663 in 27869ef

return false;

, core handles malformed transactions gracefully, and sets the inner code to MALFORMED, whereas here we'd crash.

The malformed check happens at validation, which in the parallel flow is in preParallelApply. This assert is at apply time, which you shouldn't be able to get to with an invalid transaction.

marta-lokhova · 2025-05-12T23:56:38Z

src/ledger/LedgerManagerImpl.h

@@ -175,6 +175,27 @@ class LedgerManagerImpl : public LedgerManager
        AbstractLedgerTxn& ltx,
        std::unique_ptr<LedgerCloseMetaFrame> const& ledgerCloseMeta);

+    UnorderedSet<LedgerKey> getReadWriteKeysForStage(ApplyStage const& stage);
+
+    std::unique_ptr<ThreadEntryMap> collectEntries(AppConnector& app,


There's no good reason to havegetReadWriteKeysForStage and collectEntries as LedgerManager's member functions. How about a new parallel apply utils namespace where these are free functions?

marta-lokhova · 2025-05-13T00:26:36Z

src/ledger/LedgerManagerImpl.cpp

+        threadFutures.emplace_back(std::async(
+            std::launch::async, &LedgerManagerImpl::applyThread, this,
+            std::ref(app), std::move(entryMapPtr), std::ref(cluster), config,
+            sorobanConfig, std::ref(ledgerInfo), sorobanBasePrngSeed,


A couple of things are problematic here:

std::async may throw and crash the program if the system is unable to spin up a new thread.

Passing references to threads this way is quite dangerous - if these variables go out of scope, you'll get UB. Please try to either copy or move the necessary data (e.g. ledgerInfo, cluster, etc). Long-living objects like AppConnector are probably fine, as long we can guarantee that it's destroyed after LedgerManager and any futures/shared state it's holding on to.

How does shutdown work here? If core starts shutting down while parallel apply is happening, how do we ensure threads are properly joined before the app starts to destruct?

Thinking about this more, I'm curious about the choice of std::async here as opposed to a thread pool of sorts based on the underlying hardware specs. It looks like we're not taking into account the hardware here, so I'm wondering if depending on the machine this may cause issues like CPU over-subscription. Also, this might not be an issue, but what's the rough overhead of launching new threads in std::async, since those are kernel threads?

Thinking about this more, I'm curious about the choice of std::async here as opposed to a thread pool of sorts based on the underlying hardware specs. It looks like we're not taking into account the hardware here, so I'm wondering if depending on the machine this may cause issues like CPU over-subscription. Also, this might not be an issue, but what's the rough overhead of launching new threads in std::async, since those are kernel threads?

The number of threads is determined by the network setting, which should be set based on what the network wants the minimum spec to be. Wrt launching new threads, it should be in the order of microseconds, so I don't think it makes sense to optimize with a thread pool at the moment.

std::async may throw and crash the program if the system is unable to spin up a new thread.

Isn't this similar to failing on an stl container insert due to being out of memory? In that rare instance, the node will crash, but the rest of the network should be fine.

Passing references to threads this way is quite dangerous

Yeah this is a good point. I'll work on improving this.

How does shutdown work here? If core starts shutting down while parallel apply is happening, how do we ensure threads are properly joined before the app starts to destruct?

We discussed this offline. Shutdown waits for ledger close to finish in both the background ledger close and main thread ledger close scenarios, which means these threads will have to join as well for that to happen. I can add a comment here that mentions that shutdown should work because the futures are not long lived, and we should make sure that doesn't change.

Thinking about this more, I'm curious about the choice of std::async here as opposed to a thread pool of sorts based on the underlying hardware specs. It looks like we're not taking into account the hardware here, so I'm wondering if depending on the machine this may cause issues like CPU over-subscription. Also, this might not be an issue, but what's the rough overhead of launching new threads in std::async, since those are kernel threads?

The network setting for the number of threads should be set to what we believe the minimum hardware spec should be(mentioned in the CAP. The actual number of cores used is not part of the protocol, so we have some flexibility here if we see the need to tweak this in the future.

The overhead of launching threads should be in the order of microseconds, which I don't think is something we need to optimize at the moment.

marta-lokhova · 2025-05-13T00:42:50Z

src/ledger/LedgerManagerImpl.cpp

+    std::vector<std::unique_ptr<ThreadEntryMap>> entryMapsByCluster;
+    for (auto& restoredKeysFutures : threadFutures)
+    {
+        auto futureResult = restoredKeysFutures.get();


we should add a sanity check that future is valid here (if not, calling get is UB)

Also note that after calling get, shared state isn't destroyed, because we're still storing (now invalid) futures in threadFutures. We may want to explicitly clear it to avoid footguns (thread objects are also stored in that shared state)

marta-lokhova · 2025-05-13T00:45:55Z

src/ledger/LedgerManagerImpl.cpp

+}
+
+void
+LedgerManagerImpl::applySorobanStage(AppConnector& app, AbstractLedgerTxn& ltx,


This function is doing a lot of things, making it hard to read and navigate. It might be easier to refactor each step into its own function (also please comment for each application step).

marta-lokhova · 2025-05-13T17:59:17Z

src/ledger/LedgerManagerImpl.cpp

+
+        threadFutures.emplace_back(std::async(
+            std::launch::async, &LedgerManagerImpl::applyThread, this,
+            std::ref(app), std::move(entryMapPtr), std::ref(cluster), config,


std::cref just in case (or ideally, a smart pointer to const)

Just my 5c: I don't think it's worth wrapping everything into shared ptrs here, as the lifetime is very clear (at least for the most of objects; it could make sense for something externally owned, but stuff like stage/cluster/prng seed etc. is create for the apply path and it's lifetime is managed by a single thread.
cref of course does make sense for constants.

sisuresh · 2025-05-13T20:36:24Z

src/ledger/LedgerManagerImpl.h

+        Cluster const& cluster, Config const& config,
+        SorobanNetworkConfig const& sorobanConfig,
+        ParallelLedgerInfo const& ledgerInfo, Hash const& sorobanBasePrngSeed,
+        SorobanMetrics& sorobanMetrics);


Some of the SorobanMetrics methods aren't thread safe. Those methods should assert that the thread is main or ledger close.

Alternatives to this -

Collect metrics within each thread, and update SorobanMetrics at the end.

Mark the non-thread safe member variables as atomic.

It should be fairly simple to make this thread-safe. You could also switch to medida counters, those are thread-safe.

sisuresh · 2025-05-13T22:31:59Z

src/ledger/LedgerManagerImpl.cpp

+                    results.transactionHash =
+                        txBundle.getTx()->getContentsHash();
+                    results.result = txBundle.getResPayload()->getResult();
+                    if (results.result.result.code() ==


Most of the code here is duplicated from the non-parallel path below.

sisuresh · 2025-05-21T23:21:52Z

I updated the PR after rebasing with master. The most recent rebase was messy and the PR still has a couple todo's to address from it, and some failing test cases.

sisuresh · 2025-05-21T23:27:07Z

src/ledger/LedgerTxn.cpp

@@ -825,6 +825,35 @@ LedgerTxn::Impl::erase(InternalLedgerKey const& key)
    }
 }

+void
+LedgerTxn::addRestoredFromHotArchive(LedgerEntry const& ledgerEntry, LedgerEntry const& ttlEntry)


@SirTyson restoreFromHotArchive and restoreFromLiveBucketList can be removed right? If we remove the v23 code from ApplyHelper, these will no longer be used.

Looks like the restore op uses these methods pre-v23.

src/transactions/ParallelTxEffects.h

Remove unused json test files

sisuresh · 2025-05-22T23:36:21Z

src/transactions/InvokeHostFunctionOpFrame.cpp

-    OperationResult& res,
-    std::optional<RefundableFeeTracker>& refundableFeeTracker,
-    OperationMetaBuilder& opMeta) const
+class ParallelApplyHelper


The V23 code can be removed from ParallelApplyHelper. We should also de-duplicate some of the shared code.

Did you mean from ApplyHelper? Or, rather, we need to remove pre-p23 code from this and post-p23 code from ApplyHelper.

Ah yeah I meant ApplyHelper.

dmkozh · 2025-05-23T00:36:35Z

src/ledger/LedgerManagerImpl.h

+                std::unique_ptr<ThreadEntryMap> entryMapByCluster,
+                Cluster const& cluster, Config const& config,
+                SorobanNetworkConfig const& sorobanConfig,
+                std::shared_ptr<ParallelLedgerInfo const> ledgerInfo,


nit: Why does this need to be a shared ptr, whereas soroban config is just a reference? I'd rather we passed a reference if the lifetime is clear (and it seems like it is - these threads are short-lived and are joined immediately). Or pass by value if lifetime is of concern (this is a small enough struct).

dmkozh · 2025-05-23T00:40:36Z

src/ledger/SorobanMetrics.h

-    uint64_t mLedgerInsnsCount{0};
-    uint64_t mLedgerInsnsExclVmCount{0};
-    uint64_t mLedgerHostFnExecTimeNsecs{0};
+    std::atomic<uint64_t> mCounterLedgerTxCount{0};


This is fine for the first iteration to reduce the amount of diffs, but I'd rather avoid synchronization between threads entirely and used map-reduce style approach (compute metrics per thread).

dmkozh · 2025-05-23T15:54:09Z

src/ledger/LedgerManagerImpl.cpp

-                subSeed = subSeedSha.finish();
-            }
-            ++txNum;
+            auto txSetStages = phase.getParallelStages();


auto const&

dmkozh · 2025-05-23T15:54:20Z

src/ledger/LedgerManagerImpl.cpp

@@ -1835,110 +2188,224 @@ LedgerManagerImpl::applyTransactions(

    for (auto const& phase : phases)
    {
-        for (auto const& tx : phase)
+        if (phase.isParallel())


The if/else bodies should probably be moved into separate functions.

dmkozh · 2025-05-23T15:55:32Z

src/ledger/LedgerManagerImpl.cpp

-            if (protocolVersionStartsFrom(
-                    ltx.loadHeader().current().ledgerVersion,
-                    ProtocolVersion::V_23))
+            for (size_t i = 0; i < txSetStages.size(); ++i)


This and the following loop could be range fors

dmkozh · 2025-05-23T22:44:58Z

src/transactions/InvokeHostFunctionOpFrame.cpp

+            ledgerSeq +
+            mSorobanConfig.stateArchivalSettings().minPersistentTTL - 1;
+
+        for (size_t i = 0; i < footprintKeys.size(); ++i)


Could this be range for?

dmkozh · 2025-05-23T23:05:07Z

src/transactions/InvokeHostFunctionOpFrame.cpp

+    Config const& cfg, InvokeHostFunctionOutput const& output,
+    HostFunctionMetrics const& metrics, DiagnosticEventManager& buffer) const
+{
+    if (cfg.ENABLE_SOROBAN_DIAGNOSTIC_EVENTS)


nit: Maybe return early to reduce nesting?

dmkozh · 2025-05-23T23:06:30Z

src/transactions/OperationFrame.h

@@ -62,6 +76,11 @@ class OperationFrame
    LedgerTxnEntry loadSourceAccount(AbstractLedgerTxn& ltx,
                                     LedgerTxnHeader const& header) const;

+    bool preloadEntryHelper(


Do we actually use this anywhere?

dmkozh · 2025-05-23T23:08:22Z

src/transactions/ParallelApplyStage.cpp

+}
+
+Cluster const&
+ApplyStage::getCluster(size_t i) const


nit: From what I've noticed from the code, it doesn't seem like this is necessary, given the existence of iterators (usually we only iterate this by index).

dmkozh · 2025-05-23T23:10:59Z

src/transactions/ParallelApplyUtils.cpp

+            res.emplace(lk);
+            if (isSorobanEntry(lk))
+            {
+                res.emplace(getTTLKey(lk));


Do we actually care about TTL entries for the purpose of this set? It doesn't seem like we ever actually test TTLs for being present in the set.

graydon · 2025-05-24T06:04:22Z

@sisuresh I spent today reading and understanding more of the LedgerManagerImpl changes, specifically applyThead. So far so good! but to make sense of it I also started doing some helper-extraction. I made this commit: graydon@e43974e for example. You're welcome to grab it into your branch if you think it helps readability (I do!) -- it should be a logical no-op, just method extractions and a handful of renamings or code-pattern changes (eg. using destructuring patterns in range for loops in a couple places).

if that's not too objectionable I kinda want to do the same to the other new big blocks in LedgerManagerImpl::applyTransactions and applySorobanStage. WDYT?

(it also occurs to me that most of these methods don't touch LedgerManagerImpl state variables at all and could perhaps be moved to a separate file. LedgerManagerImpl is fairly large at this point..)

sisuresh · 2025-05-27T18:12:15Z

some

Thank you for this! Your change looks great and I'll cherry-pick it. The other big blocks should be split up as well. If that's something you wanted to tackle, that would be great.

graydon · 2025-05-27T18:17:50Z

some

Thank you for this! Your change looks great and I'll cherry-pick it. The other big blocks should be split up as well. If that's something you wanted to tackle, that would be great.

@sisuresh did another quick round of this in graydon@8207c92

sisuresh · 2025-05-27T20:26:33Z

src/ledger/LedgerManagerImpl.cpp

@@ -2526,7 +2526,8 @@ LedgerManagerImpl::sealLedgerTxnAndTransferEntriesToBucketList(
                    LiveBucket::FIRST_PROTOCOL_SUPPORTING_PERSISTENT_EVICTION))
            {
                std::vector<LedgerKey> restoredKeys;
-                auto const& restoredKeyMap = ltx.getRestoredHotArchiveKeys();
+                auto const& restoredKeyMap =
+                    ltxEvictions.getRestoredHotArchiveKeys();


This change doesn't actually work because getRestoredHotArchiveKeys only returns mRestoredKeys for the current ltx, not its parents. We should probably recurse up and append the keys instead. @SirTyson What do you think?

sisuresh commented May 7, 2025

View reviewed changes

src/testdata/ledger-close-meta-enable-classic-events-v1-protocol-23-soroban.json Outdated Show resolved Hide resolved

anupsdf requested a review from Copilot May 7, 2025 23:00

Copilot AI reviewed May 7, 2025

View reviewed changes

src/transactions/FeeBumpTransactionFrame.cpp Outdated Show resolved Hide resolved

src/ledger/LedgerTxn.cpp Show resolved Hide resolved

sisuresh force-pushed the par-rebase branch from 1a1ca23 to a5d85bd Compare May 7, 2025 23:35

sisuresh commented May 7, 2025

View reviewed changes

src/ledger/LedgerManagerImpl.cpp Outdated Show resolved Hide resolved

sisuresh force-pushed the par-rebase branch 3 times, most recently from 7402b61 to 602914b Compare May 8, 2025 16:07

sisuresh requested review from graydon, marta-lokhova and dmkozh May 8, 2025 17:07

sisuresh commented May 8, 2025

View reviewed changes

src/ledger/LedgerManagerImpl.cpp Outdated Show resolved Hide resolved

marta-lokhova reviewed May 9, 2025

View reviewed changes

sisuresh commented May 9, 2025

View reviewed changes

src/transactions/TransactionFrameBase.h Outdated Show resolved Hide resolved

sisuresh commented May 9, 2025

View reviewed changes

src/ledger/LedgerManagerImpl.cpp Outdated Show resolved Hide resolved

sisuresh force-pushed the par-rebase branch from 704359d to 167bf21 Compare May 9, 2025 22:51

sisuresh commented May 9, 2025

View reviewed changes

marta-lokhova requested changes May 13, 2025

View reviewed changes

marta-lokhova reviewed May 13, 2025

View reviewed changes

sisuresh commented May 13, 2025

View reviewed changes

sisuresh force-pushed the par-rebase branch from 56ea220 to 02d8769 Compare May 21, 2025 23:20

sisuresh commented May 21, 2025

View reviewed changes

src/transactions/ParallelTxEffects.h Outdated Show resolved Hide resolved

sisuresh force-pushed the par-rebase branch 2 times, most recently from f6318fd to cbb3acf Compare May 22, 2025 15:27

sisuresh added 2 commits May 22, 2025 18:11

Parallelize Soroban

5425666

Stop throwing exceptions on unrecoverable errors

83e960f

sisuresh added 2 commits May 22, 2025 18:20

Update tests

2098c7a

Remove unused json test files

make format

56c4fe8

sisuresh force-pushed the par-rebase branch from cbb3acf to 56c4fe8 Compare May 22, 2025 23:20

sisuresh commented May 22, 2025

View reviewed changes

sisuresh added 2 commits May 23, 2025 10:54

Ltx fixes

9760cac

Fix test

678ac8b

dmkozh reviewed May 23, 2025

View reviewed changes

sisuresh commented May 27, 2025

View reviewed changes

Parallelize Soroban #4715

Are you sure you want to change the base?

Parallelize Soroban #4715

Uh oh!

Conversation

sisuresh commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sisuresh May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marta-lokhova May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sisuresh commented May 7, 2025 •

edited

Loading

sisuresh May 9, 2025 •

edited

Loading

marta-lokhova May 13, 2025 •

edited

Loading

sisuresh commented May 21, 2025 •

edited

Loading

sisuresh May 21, 2025 •

edited

Loading