Hi, Sorry for my last pict attachment. there is still filter transform process before produce to kafka. when i check it again, if i use rebalance function, there will be stuck on the first process after consume message. on my last post. the process stuck on filter process. After consume from topic, the messages sent to filter transform. but it never sent to the next process. i've try to make the simple process. read from kafka topic and then produce it into another topic. when using rebalance function, the message success to produce. So my question is how can the messages stuck on the first process after consume from the topic ? and why they never sent to the next process ???
[image: image.png] On Mon, Nov 30, 2020 at 6:01 PM Youzha <yuza.ras...@gmail.com> wrote: > Hi Arvid, > > thanks for your reply. > > yes i’m using exactly once mode and i’ve enable checkpointing. > > env.enableCheckpointing(20000, CheckpointingMode.EXACTLY_ONCE); > > > this is my flow in the web UI. i’m trying to do the simple thing. read > from kafka and then sink it to another topic. > > > this is the web UI flow > > > > there are no msg produce on my new one topic. > > > any suggestion about this? 🙏🏻 > > On Mon, 30 Nov 2020 at 17.10 Arvid Heise <ar...@ververica.com> wrote: > >> Hi Youzha, >> >> do you use exactly once mode in kafka producer? Make sure that you have >> enabled checkpointing and set the interval appropriately. >> >> You can also see the message flow in the web UI. Check if something is >> reaching the sink. Aside from that, if you use window operators make sure >> that they fire. >> >> In general, the information that you provided is very sparse. I can give >> more detailed pointers with more information. >> >> On Mon, Nov 30, 2020 at 9:25 AM Youzha <yuza.ras...@gmail.com> wrote: >> >>> Hi, i’m using kafka consumer and producer on flink. >>> >>> i’ve succeed to consume, transform, and produce to another topic on my >>> development env ( 1 partition and a few dummy msg) >>> >>> but when i try to submit job on production env ( 20 partitions and the >>> size is about 100Gb ), >>> there is no one message produce into the kafka topic. i’ve try to set >>> flink parallelism into 20 but is still not working. then i try to use >>> rebalance function it’s success to consume the messages but it still no one >>> msg produce into kafka topic. >>> >>> pls advice >>> >> >> >> -- >> >> Arvid Heise | Senior Java Developer >> >> <https://www.ververica.com/> >> >> Follow us @VervericaData >> >> -- >> >> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >> Conference >> >> Stream Processing | Event Driven | Real Time >> >> -- >> >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g> >> >> -- >> Ververica GmbH >> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >> (Toni) Cheng >> >
