[do not merge] Local performance test of ingestion #3292
Conversation
Signed-off-by: Oriol Muñoz <[email protected]>
Signed-off-by: Oriol Muñoz <[email protected]>
Signed-off-by: Oriol Muñoz <[email protected]>
There was a problem hiding this comment.
ignore this file for now
There was a problem hiding this comment.
downloads 1h worth of mainnet updates and dumps it into a file, to be loaded in the test above
| .readString(dumpFile) | ||
| .replace( // TODO: maybe just use the store with dsoParty=mainnet's | ||
| "DSO::1220b1431ef217342db44d516bb9befde802be7d8899637d290895fa58880f19accc", | ||
| dsoParty.toProtoPrimitive, |
There was a problem hiding this comment.
every test uses its own dsoParty... see TODO
| store.multiDomainAcsStore.ingestionSink.initialize().futureValue | ||
| val timings = mutable.ListBuffer[Long]() | ||
| val ingestionConfig = IngestionConfig() | ||
| val dumpFile = Paths.get("/home/oriolmunoz/mainnetdump/update_history_response.json") |
There was a problem hiding this comment.
this dump is 1h of mainnet updates (see next file for how it was obtained).
It should be okay, because:
- Any (filtered-in) create causes an insert
- Any archive causes a delete, so as long as the template is included in the filter
- Txlog doens't care about whether the contract ids involved are present in the ACS store or not
|
@rautenrieth-da can I ask you to verify that what I did is not nonsense? Ignore the fact it's quick & dirty, just trying to make sure that I'm not missing something important here. |
| val duration = after - before | ||
| timings ++= Seq.fill(batch.length)(duration / batch.length) | ||
| val avg = timings.sum.toDouble / timings.size | ||
| println( |
There was a problem hiding this comment.
Does this wait for the output to be flushed? If yes, it could add significant overhead, consider only printing at the end or only once every N iterations.
...org/lfdecentralizedtrust/splice/performance/tests/DbSvDsoStoreIngestionPerformanceTest.scala
Outdated
Show resolved
Hide resolved
| store.multiDomainAcsStore.ingestionSink | ||
| .ingestUpdateBatch( | ||
| NonEmptyList.fromListUnsafe( | ||
| txs |
There was a problem hiding this comment.
Isn't this ingesting the whole txs list in each iteration? Shouldn't it use batch instead? I'm surprised nothing explodes when we reingest the same transactions over and over.
There was a problem hiding this comment.
🤦 I'm surprised my performance was not god-awful
There was a problem hiding this comment.
It was probably fine because the first iteration did a lot of work, and all other iterations were no-ops due to the last ingested offset guard.
Signed-off-by: Oriol Muñoz <[email protected]>
Signed-off-by: Oriol Muñoz <[email protected]>
Quick & dirty, but good enough to test locally