Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize db.insert_multiple(points) Execution Time #54

Open
Aminmalek opened this issue Dec 14, 2024 · 0 comments
Open

Optimize db.insert_multiple(points) Execution Time #54

Aminmalek opened this issue Dec 14, 2024 · 0 comments

Comments

@Aminmalek
Copy link

Is your feature request related to a problem? Please describe.
The current implementation of db.insert_multiple(points) takes approximately 2 minutes to execute when handling a large number of points. This duration significantly impacts performance and scalability, especially in scenarios requiring frequent batch insertions.

Describe the solution you'd like
I would like to optimize the insert_multiple method to reduce its execution time. Possible solutions could include:

Implementing multithreading or multiprocessing to parallelize the insertion process.
Adding batch processing logic to divide the points into smaller, more manageable chunks that can be processed concurrently.
Leveraging database-specific bulk insertion features to improve performance.

Describe alternatives you've considered
Using a single-threaded approach with smaller batch sizes. However, this still does not fully utilize available resources, and performance gains are marginal.
Splitting the workload manually outside the function, but this introduces additional complexity and redundancy in code.

Additional context
Screenshot From 2024-12-14 18-16-38

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant