Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization of NOC Layer's put_object Operation #194

Open
lurenpluto opened this issue Apr 11, 2023 · 1 comment
Open

Optimization of NOC Layer's put_object Operation #194

lurenpluto opened this issue Apr 11, 2023 · 1 comment
Labels
Any Idea Any problem/ideas/suggestions feature New feature Performance about performance issues

Comments

@lurenpluto
Copy link
Member

During the testing process of the isolate-path-env in #159, it has been found that the performance is poor when a group of objects is saved using noc.put_object within the cache.commit stage, especially when the internal write operations in op-env lead to a switch to hub mode for the object-map. This mode switch unavoidably generates a large number of sub object-maps, resulting in a time-consuming commit process.

Therefore, future optimizations should be considered.

@lurenpluto
Copy link
Member Author

Below are a few possible optimization points currently under consideration:

  • Add batch operations in the noc interface
    This is mainly for batch put_object operations, and the corresponding blob and meta layers also need to be optimized for batch processing.

  • Optimize the blob storage structure
    This optimization is not only for improving write performance but also for read and storage improvements. Currently, objects are stored as individual files in the objects directory, which can lead to excessive disk files and performance issues when there are a massive number of objects. Consider using a pack-based storage approach or other similar methods to address this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Any Idea Any problem/ideas/suggestions feature New feature Performance about performance issues
Projects
Status: 💬To Discuss
Development

No branches or pull requests

1 participant