Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Java] Memory issue about FuryInputStream #1576

Open
yihwang6 opened this issue Apr 26, 2024 · 4 comments
Open

[Java] Memory issue about FuryInputStream #1576

yihwang6 opened this issue Apr 26, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@yihwang6
Copy link

In #1551, I noticed that a growBuffer method was added to FuryInputStream, which continuously increases the size of the buffer. This operation may potentially give rise to memory-related concerns due to the large buffer size. It might be better to consider implementing the buffered streaming reading with a fixed-size buffer.

@yihwang6 yihwang6 added the enhancement New feature or request label Apr 26, 2024
@chaokunyang
Copy link
Collaborator

org.apache.fury.io.FuryInputStream#shrinkBuffer will be invoked after the deserialziation finished

@yihwang6
Copy link
Author

Ok, but there may still be a large buffer size when deserializing a large object.

@chaokunyang
Copy link
Collaborator

Yes, that is possible. We don't release buffer before we finished the deserialization. One case I can imagine is that when buffer close to int max size, we can't multiple 1.5.

@chaokunyang
Copy link
Collaborator

Would you like to help optimize this?

chaokunyang added a commit that referenced this issue Apr 27, 2024
## What does this PR do?

grow buffer to interger.max

## Related issues

#1576 


## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/incubator-fury/issues/new/choose)
describing the need to do so and update the document if necessary.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?


## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants