Improve Performance For "MaxLineSize" and Concurrency of ParseString(line)

[lhanjian]

prometheus-nginxlog-exporter/tail/tailer.go

Line 44 in f09c9dc

Poll: true,

add MaxLineSize: 65534 to improve Throughput of reading file?

prometheus-nginxlog-exporter/main.go

Line 372 in f09c9dc

for line := range t.Lines() {

to Concurrent Execute ParseString(line) in this loop?

over 25000 QPS access.log, exporter unable to update metrics accurately.

Tks.

[martin-helmich]

25k qps is a respectable workload 😳 -- admittedly, this is the heaviest workload that I've seen this exporter been subjected to. I'd be happy to include optimizations for such heavy workloads, however I can't make any promises as to when I'll actually find the time to get to that. In the meantime, I'm happy to accept any PRs in this matter. 🙂

[gburton1]

Do you have a rule of thumb for the max qps that you know the exporter can handle without pegging the one core that it uses and falling behind on log processing?