Skip to content

Performance degradation on python producer.send with large number of records #258

Open
@AlexMikhalev

Description

@AlexMikhalev

slack thread: https://infinyon.slack.com/archives/CQ8UX6SBH/p1684349700779719

simple producer in python:

#!/usr/bin/env python3
from datetime import datetime
from fluvio import Fluvio
import time 

PARTITION = 0

if __name__ == "__main__":
   # Connect to cluster
   fluvio = Fluvio.connect()

   producer = fluvio.topic_producer("simple")

   start_time = time.time()

   for x in range(10000):
       producer.send_string("{}: timestamp: {}".format(x, datetime.now()))

   # Flush the last entry
   producer.flush()
   print("completed in  {} seconds", (time.time()-start_time))

10K records pushed in 1.2153611183166504 seconds, when if you increase the range to 100000
python3 producer100K.py
The same code takes 64.67606687545776 to complete.
Rust producer doesn't have such behaviour.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions