Hi, I've been trying to set up pulsar(2.4.2) with flink(1.8.2) to run window transformations on data being streamed to pulsar topics.
I have run into a few issues which I worked around; however I'm not sure if
what I've done is the intended way it's supposed to work, or it's a bug or if
I'm missing some configurations.
These are my observations so far :
Setup :
- flink 1.8.2
- pulsar 2.4.2
- a flink job with a sliding window transformation
- using pulsar-flink libraries 2.4.2
- flink parallellism set to 4
- to start with, my pulsar topics had a single partition.
Observations:
- When I set the stream characteristic to processing time, everything works
fine. I observe transformed data in the destination topic.
- When I set the stream characteristic to event time, I do not observe any data
in the destination topic.
- On closer inspection, I found that the onEventTime() of the trigger does not
get called.
- After some debugging, I notice that :
- when I set
consumerConfigurationData.setSubscriptionType(SubscriptionType.Shared);, it
works
- when I set
consumerConfigurationData.setSubscriptionType(SubscriptionType.Failover); it
does not work.
- I also noticed that, when I increase the number of partitions on my source
and sink topics to 4, it works even with SubscriptionType.Failover
- However if I have fewer partitions than the parallelism that I have set in
flink, it stops working.
- I would like to understand the reason for this above behaviour; is it a bug,
or is there some other configuration I am missing, and how I can get it working
the correct way
- SubscriptionType.Shared, as I understand, does not ensure that data will be
processed in order, so I am unsure if I can set that.
- I also don't think having to set the partition count equal to the parallelism
is a solution, for us.
I am also attaching the flink code that I am using.
Thank you in advance for your help !
Thanks,
Subbu
<<attachment: pulsar-flink-preload-new.zip>>
