Re: problem with kafka on multiple partitions and large size message

2020-11-30 Thread Youzha
first process after consume from the topic ? and why they never sent to the next process ??? [image: image.png] On Mon, Nov 30, 2020 at 6:01 PM Youzha wrote: > Hi Arvid, > > thanks for your reply. > > yes i’m using exactly once mode and i’ve enable checkpointing. > > env

Re: problem with kafka on multiple partitions and large size message

2020-11-30 Thread Youzha
web UI flow there are no msg produce on my new one topic. any suggestion about this? 🙏🏻 On Mon, 30 Nov 2020 at 17.10 Arvid Heise wrote: > Hi Youzha, > > do you use exactly once mode in kafka producer? Make sure that you have > enabled checkpointing and set the interval appropriat

problem with kafka on multiple partitions and large size message

2020-11-30 Thread Youzha
Hi, i’m using kafka consumer and producer on flink. i’ve succeed to consume, transform, and produce to another topic on my development env ( 1 partition and a few dummy msg) but when i try to submit job on production env ( 20 partitions and the size is about 100Gb ), there is no one message produ

left join flink stream

2020-11-16 Thread Youzha
Hi i want to do join reference between kafka with mysql table reference. how can i do this thing with flink stream. does coGroup function can handle this ? or anyone have java sample code with this case? i’ve read some article that said if cogroup function can do left outer join. but i’m still conf

split avro kafka field

2020-11-16 Thread Youzha
hi, i’am a new comer to learn flink stream. and i wanna try to split the kafka avro field into multiple fields. for example i have this one in my kafka topic : { id : 12345, name : “john”, location : “indonesia;jakarta;south jakarta” } and then i wanna split the location value by “;” in