feat: refactor persistent term usage to behaviour #344
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds in an ETS configuration storage option, for #342 .
Our use case has many dynamically-created Broadway servers (each with different non-static names). This results in global GC getting triggered. Although fb69a7a has some improvements, it is insufficient for our situation.
If we were to continue using persistent term using the above mentioned change, it will result in large buildup in the hash table over time. As time to update/store the term is proportional to hash table size, resulting in degrading storage speeds over time. In our case, the memory buildup over time may also be non-negligible.
However, without the above change (where the persistent term is not cleaned up), it will result in the GC getting triggered each time the server goes down (due to our autoscaling logic).
Switching over to
:ets
based storage would avoid the GC-ing issues.