Apache Beam support for Pulsar

[svenhornberg]

I would like to use Apache Beam and Pulsar as my source, for writing my batch or streaming jobs.

Please consider adding a PulsarIO class to the sdk like Kafka (https://beam.apache.org/releases/javadoc/2.4.0/org/apache/beam/sdk/io/kafka/KafkaIO.html)

I found the pulsar-flink and pulsar-spark folders, but i really like the Apache Beam API.
Maybe you could add a pulsar-beam example or subfolder

[aahmed-se]

this should be a request in the beam repo.

[svenhornberg]

okay, looks like there is a story for that (https://issues.apache.org/jira/browse/BEAM-8218), but anyway it would be a push for pulsar. Maybe working together on this is an option.

[sijie]

@svenhornberg there is a discussion in BEAM mailing list. We will collaborate with beam community on adding a pulsar connector in BEAM repo.

[svenhornberg]

Any statement if you would try to implement it ? 3 months have passed

[sijie]

@svenhornberg The beam community was trying to drive the development for this connector. Since they are the experts of beam. I didn't closely follow the progress there. Happy to follow up with them and get back here.

[svenhornberg]

@sijie thank you for clarifying

[rfuerst87]

Any updates on this matter? Would be interested in a Beam Connector too :)

[sijie]

@rfuerst87 - the beam community is doing the work. You can track the progress here - https://issues.apache.org/jira/browse/BEAM-8218

I will follow up there as well.

[rfuerst87]

@sijie Thanks for the update. Thought you might have some more insights from the mailing list. Will follow the topic in JIRA.

[alexvanboxel]

I've taken over the ticket. I'm familiar with Beam, less so with Pulsar (I want to learn Pulsar, creating the IO). I'll be starting locally with Pulsar in standalone mode, as soon as I have a prototype I'll keep updates on this thread.

[sijie]

@alexvanboxel awesome! much appreciated your help!

You can also check how Pulsar was integrated with Spark and Flink as a reference. The pulsar-flink one is the one is contributed to upstream Flink as FLIP-72 and

https://github.com/streamnative/pulsar-spark
https://github.com/streamnative/pulsar-flink

@yjshen can help answer any questions about such integrations.

[alexvanboxel]

Great, I appreciate the help. I can happily report that after a few hours I already got some basic Google Cloud Pubsub -> Apache Beam -> Pulsar pipeline working (on the local runner). Don't get too excited to, still a lot of work to be done.

I'll have a look at the integrations.

[chethanuk]

Apache Beam -> Pulsar pipeline working

Cool, :)

[adrianog]

Any update?

[hpvd]

when looking into beams i/o connector matrix pulsar is not listed yet:
https://beam.apache.org/documentation/io/connectors/

[hpvd]

but there is a sdk package
Package org.apache.beam.sdk.io.pulsar
https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/pulsar/package-summary.html

[hpvd]

to get an idea of the current state, there are some open issues in beam for pulsar:
https://github.com/apache/beam/issues?q=is%3Aissue+pulsar+is%3Aopen

[hpvd]

in v2.39 of beam a relatively basic pulsar io was introduced:
Implemented Apache PulsarIO https://beam.apache.org/blog/beam-2.39.0/
for details see:
https://issues.apache.org/jira/browse/BEAM-8218

[vcastell-tibco]

I see that a Pulsar connector was made available as of BEAM 2.38.0 release but I don't see Pulsar as an official connector on below page. Is the Pulsar IO connector official or not? If official then can someone please update the page since it gives idea that a Pulsar IO connector is not available.

https://beam.apache.org/documentation/io/connectors/

[hpvd]

just opened an issue in Beam repository to maintain a good overview on this:
apache/beam#31078

[hpvd]

opened an issue to add current state of Pulsar support to Beams doc:
apache/beam#31079

[hpvd]

the progress for pulsar on becoming a first class citizen in beam, seems to have stucked :-(

There are not only some

• open issues, but also
• non merged but automatically as stale closed pull request.

Have integrated them into the overview: apache/beam#31078

[hpvd]

if you are interested in Beam support for Pulsar plz upvote this discussion topic to gain visibility
and take a look on open issues in beam repository summarized in apache/beam#31078