A more full featured Pulsar Flink Connector can be found here: https://github.com/streamnative/pulsar-flink
On Fri, Dec 13, 2019 at 1:43 PM Jerry Peng <jerry.boyang.p...@gmail.com> wrote: > > Hello Subbu, > > Responding to your comments in Line: > > > When I set the stream characteristic to event time, I do not observe any > > data in the destination topic. > > Event time is currently not support in this version of of the source > > > I would like to understand the reason for this above behavior; is it a bug, > > or is there some other configuration I am missing, and how I can get it > > working the correct way > > I also noticed that, when I increase the number of partitions on my source > > and sink topics to 4, it works even with SubscriptionType.Failover > > When using FAILOVER subscription, the parallelism of your source must > be equal to or less than the number of partitions of the topic the > source is reading from. If the parallelism of your source is less > than the number of partitions, then there will be source instances > that will be assigned to read from one or more partitions > > > SubscriptionType.Shared, as I understand, does not ensure that data will be > > processed in order, so I am unsure if I can set that. > > That is correct. Using a Failover subscription will maintain > processing order relative to particular partition. > > Best, > > Jerry > > Best, > > Jerry > > On Fri, Dec 13, 2019 at 8:25 AM Subramanyam Ramanathan > <subramanyam.ramanat...@microfocus.com> wrote: > > > > Hi, > > > > > > > > I've been trying to set up pulsar(2.4.2) with flink(1.8.2) to run window > > transformations on data being streamed to pulsar topics. > > > > > > > > I have run into a few issues which I worked around; however I'm not sure if > > what I’ve done is the intended way it’s supposed to work, or it’s a bug or > > if I'm missing some configurations. > > > > > > > > These are my observations so far : > > > > > > > > Setup : > > > > - flink 1.8.2 > > > > - pulsar 2.4.2 > > > > - a flink job with a sliding window transformation > > > > - using pulsar-flink libraries 2.4.2 > > > > - flink parallellism set to 4 > > > > - to start with, my pulsar topics had a single partition. > > > > > > > > Observations: > > > > - When I set the stream characteristic to processing time, everything works > > fine. I observe transformed data in the destination topic. > > > > - When I set the stream characteristic to event time, I do not observe any > > data in the destination topic. > > > > - On closer inspection, I found that the onEventTime() of the trigger does > > not get called. > > > > - After some debugging, I notice that : > > > > - when I set > > consumerConfigurationData.setSubscriptionType(SubscriptionType.Shared);, it > > works > > > > - when I set > > consumerConfigurationData.setSubscriptionType(SubscriptionType.Failover); > > it does not work. > > > > - I also noticed that, when I increase the number of partitions on my > > source and sink topics to 4, it works even with SubscriptionType.Failover > > > > - However if I have fewer partitions than the parallelism that I have set > > in flink, it stops working. > > > > - I would like to understand the reason for this above behaviour; is it a > > bug, or is there some other configuration I am missing, and how I can get > > it working the correct way > > > > - SubscriptionType.Shared, as I understand, does not ensure that data will > > be processed in order, so I am unsure if I can set that. > > > > - I also don't think having to set the partition count equal to the > > parallelism is a solution, for us. > > > > > > > > I am also attaching the flink code that I am using. > > > > > > > > Thank you in advance for your help ! > > > > > > > > Thanks, > > > > Subbu > > > > > > > > > > > >
