Description
Hi all,
I have following problem when sending data points to InfluxDB using asynchronous WriteApi.
The problem occurs only when processing data points that should be sent to different InfluxDB buckets
in parallel. In that case, POST request to InfluxDB is never sent and client doesn't generate any
error (WriteErrorEvent, WriteRetriableErrorEvent, ...). The only way how we detect that data is not sent
to InfluxDB is the fact that WriteSuccessEvent is never published.
Our application receives list of events using REST API, processes these events (processing is simple and fast) and transfers each event to InfluxDB point.
Data points are relatively simple:
- max 5 tags
- 1 field
The problem occurs when we're processing multiple requests at same time:
- Request A - has more than ~500 points, sending points to bucket A
- Request B - has more than ~500 points, sending points to bucket B
Processing of both requests starts at near same time (HTTP request is received at near same time) and we process all points in parallel (multiple threads).
We use following method for sending data point to InfluxDB (we're sending point by point):
- writeApi.writePoint(bucket, organisation, point);
We use default WriteOptions (batch size = 1000, max number of retries = 3, ...)
Steps to reproduce:
- Create at least two buckets in InfluxDB
- Create Java application
- Thread A will process request A (containing > 500 data points that need to be sent to bucket A)
- Thread B will process request B (containing > 500 data points that need to be sent to bucket B)
- Threads A and B will start at near same time
- Both threads will process data point by starting new Thread for processing each point (for test, there can be thread pool with 1000 threads for processing events)
- Each "event processing thread" will block until it receives WriteSuccessEvent - we have to be sure that point is written to InfluxDB
Also, if we decrease number of threads that process events in parallel to 50, then everything works as expected. When we have more than 100 threads, processing is stuck.
Expected behavior:
All created points written to InfluxDB
Actual behavior:
Points are not written to InfluxDB. POST request for writing points to InfluxDB is not created. Since POST request is never generated (like batch is stuck), there is no Event published that could tell if writing was successful or there was some error.
Specifications:
- Client Version: 1.15
- InfluxDB Version: 2.0.2
- JDK Version: openjdk 11.0.9
- Platform: CentOS 8
Please let me know if you need any additional information.
Thank you and best regards,
Domagoj