Refactor indexing primitives and optimize bulk creation #56
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Removed events parameter from `indexPrimitives` and migrated related logic to `indexingHandler` for better modularity. - Improved `optimizedBulkCreate` to support higher throughput with parallel writes and increased batch size. - Reduced verbose logging across multiple files to enhance output clarity. - Added a new script `subquery-publish.sh` for seamless environment-specific publish automation. - Small fixes: cleaned up imports and removed unused debug statements for better readability.
We were trigger a lot of promises in parallel, and for some reason when we do this, Sequelize in the background hangs the transactions. Lowering the number of parallel promises looks like giving more free space into the event loop, which allows To Sequelize to do his job. - Reordered asynchronous operations in `indexer.manager.ts` for better handling of event processing. - Introduced a new function `sanitize_cmd` in `shared.sh` to redact sensitive command-line values for logging. - Updated default values for `BATCH_SIZE` and `CONCURRENCY` in `db.ts`, and added a log entry for global config parameters. - Enhanced `node-entrypoint.sh` to include additional environment variables and sanitized command logging for execution safety. - Fixed minor formatting issues in `project.ts` handler configuration.
- Temporarily commented out an ARM64 platform in workflow matrix due to excessive build times. - Added a note explaining the issue and the need for further investigation. - Ensured that the AMD64 build process remains unaffected.
Alann27
approved these changes
Jun 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
indexPrimitivesand migrated related logic toindexingHandlerfor better modularity.optimizedBulkCreateto support higher throughput with parallel writes and increased batch size.Issue
There are blocks on Beta with a size over 700mb (
rpc->/block_results?height=X)One of those massive blocks contains 1.5 million of events, which need to be filter, processed and indexed.
All this is taking way more, like 2 or 3 minutes on some cases even with transaction, batching, 3 servers with 16 cores and 128gb Ram, NVME disks and full tuneup PostgresQL.
Type of change
Select one or more:
Sanity Checklist