Asynchronous write fails when writing data to multiple buckets #202
Comments
|
@dmajic, thanks for using our client and your description, we will take a look. |
|
Hi @dmajic, Just for clarification... The "event processing thread" calls Regards |
Hi @bednar , correct. All threads that call writePoint method are "hooked to WriteSuccessEvent". BR, |
|
Hi @bednar , could you please send update regarding this issue? Have you managed to reproduce described behaviour? Thank you and best regards, |
|
@bednar I found the same problem. When multiple threads write batches of data to two buckets at the same time, the data cannot be written successfully. I try to use different writeApi instance for different buckets, and the writing can be successful at the same time。 |
|
@bednar I found the same problem. I used singleton instance of WriteApi to write in two different buckets. Data is written properly for some indefinite period(sometime 7 hours, sometime 15 minutes etc.) then stops automatically. There was no error or success event called. If I restart my application(JVM) it starts writting again. Please let me know what is the way forward. I am using influxdb-client-java v 4.1.0. |
|
I think I have the same problem, with 6.0.0. But I also get an Exception, and it happens in just a minute:
|
|
I got the same problem with 6.0.0, and find a issues in RxJava #7100. |
|
Hi @White-Lee, how looks your code to ingesting data into InfluxDB? How many data your ingestor produces? Regards |
|
Hi bednar, Test code like this. Hope this will help you. WriteApi writer = influxDBClient.makeWriteApi();
new Thread(() -> {
while (true) {
List<Point> points = new ArrayList<>();
for (int i = 0; i < 129; i++) {
points.add(Point.measurement("xxx")
.addField("id", i)
.addTag("tag" + i, i + "")
.time(System.currentTimeMillis(), WritePrecision.MS));
}
List<String> s = points.stream().map(Point::toLineProtocol).collect(Collectors.toList());
writer.writeRecords(WritePrecision.MS, s);
try {
sleep(805);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}).start();
new Thread(() -> {
while (true) {
List<Point> points = new ArrayList<>();
for (int i = 0; i < 129; i++) {
points.add(Point.measurement("xxx")
.addField("id", i)
.addTag("tag" + i, i + "")
.time(System.currentTimeMillis(), WritePrecision.MS));
}
List<String> s = points.stream().map(Point::toLineProtocol).collect(Collectors.toList());
writer.writeRecords(WritePrecision.MS, s);
try {
sleep(759);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}).start();The problem appeared after the test program ran for 2 hours. stack Specifications: Client Version: 6.0.0 |
|
|
@linghengqian, @White-Lee Thank you for the detailed information. We will take a look ASAP. |
|
I can reproduce this same issue with 6.0.1 by simply stopping the remote instance of InfluxDB while writes are ongoing. I have a simple Java application running in docker that periodically writes data to a single bucket in a single thread. If I stop the InfluxDB docker container while this application is running, I see the same issue. The write API does not recover if the InfluxDB container is restarted. Client: 6.0.1, InfluxDB: 2.1, JDK: 17.0.2, Platform: Docker Relevant log output: |
|
Hi All, I am currently working on this issue and I hope so that I found the solution. I've prepared a fixed version of the client in the #358. The development version is deployed into Any feedback will be appreciate 👂 Regards |
|
@bednar This fixes my issue (stopping influx server while writes are ongoing). The write API successfully recovers and resumes writing after the influx server is restarted. Thank you! |
|
Hi bednar, |
|
@robertjgtoth, @White-Lee thanks for your feedback 👍 We have to implement additional test cases before we will be ready merge it in |
|
I tested |
|
I have same problem, my influxdb client lost data after startup and write data for 2 hours, and report "Unable to emit a new group (#2) due to lack of requests". When I using 6.2.0.groupBy-SNAPSHOT, this issue still exists. please check if anything wrong. Client: 6.2.0.groupBy-SNAPSHOT, pom.xml, code and output log is attached; thanks. related log: |
|
@linghengqian thanks for testing |
|
Hi All, The PR #358 is ready to review before merge into master. The development version is deployed into Any feedback and testing will be appreciate👂 Regards |
Hi all,
I have following problem when sending data points to InfluxDB using asynchronous WriteApi.
The problem occurs only when processing data points that should be sent to different InfluxDB buckets
in parallel. In that case, POST request to InfluxDB is never sent and client doesn't generate any
error (WriteErrorEvent, WriteRetriableErrorEvent, ...). The only way how we detect that data is not sent
to InfluxDB is the fact that WriteSuccessEvent is never published.
Our application receives list of events using REST API, processes these events (processing is simple and fast) and transfers each event to InfluxDB point.
Data points are relatively simple:
The problem occurs when we're processing multiple requests at same time:
Processing of both requests starts at near same time (HTTP request is received at near same time) and we process all points in parallel (multiple threads).
We use following method for sending data point to InfluxDB (we're sending point by point):
We use default WriteOptions (batch size = 1000, max number of retries = 3, ...)
Steps to reproduce:
Also, if we decrease number of threads that process events in parallel to 50, then everything works as expected. When we have more than 100 threads, processing is stuck.
Expected behavior:
All created points written to InfluxDB
Actual behavior:
Points are not written to InfluxDB. POST request for writing points to InfluxDB is not created. Since POST request is never generated (like batch is stuck), there is no Event published that could tell if writing was successful or there was some error.
Specifications:
Please let me know if you need any additional information.
Thank you and best regards,
Domagoj
The text was updated successfully, but these errors were encountered: