You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In my scenario, I have 80000 messages per second and I have to store in elasticsearch/opensearch which will run continuously for months.
Setup: 3 nodes(12 CPU, 128GB RAM) deployed in same ESXI host. opensearch is interconnected between these nodes.
Actual Process: 1 node will receive the docs in the form of UDP packets and distribute it with other 2 nodes via forward plugin to store it in opensearch.
ISSUE we are facing is, while storing it in opensearch in few minutes we are seeing slow flush log thrushold warning and read timed out. it is taking several retry attempts to store the documents in opensearch and some of the events are dropped in pipeline.
Questions:
Considering the above setup, what will be the approximate throughput index value in opensearch?
if the nodes are deployed in different ESXI host forward plugin drops some the data, we are suspecting that it is struggling to reload the connection between the nodes. Is there any way to debug this to know what exactly happening during this transaction?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
In my scenario, I have 80000 messages per second and I have to store in elasticsearch/opensearch which will run continuously for months.
Setup: 3 nodes(12 CPU, 128GB RAM) deployed in same ESXI host. opensearch is interconnected between these nodes.
Actual Process: 1 node will receive the docs in the form of UDP packets and distribute it with other 2 nodes via forward plugin to store it in opensearch.
ISSUE we are facing is, while storing it in opensearch in few minutes we are seeing slow flush log thrushold warning and read timed out. it is taking several retry attempts to store the documents in opensearch and some of the events are dropped in pipeline.
Questions:
Beta Was this translation helpful? Give feedback.
All reactions