Re: Please suggest helpful tools

2020-01-13 Thread Eva Eva
d this but noticed that it didn't work as the data skew (and heavy load on one task) continued. Could you please let me know if I missed anything? Thanks, Eva On Sun, Jan 12, 2020 at 8:44 PM Kurt Young wrote: > Hi, > > You can try to filter NULL values with an explicit condition like

Re: Please suggest helpful tools

2020-01-10 Thread Eva Eva
;> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/checkpoint_monitoring.html >> [2] >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/back_pressure.html >> >> Best >> Yun Tang >> --

Please suggest helpful tools

2020-01-09 Thread Eva Eva
Hi, I'm running Flink job on 1.9 version with blink planner. My checkpoints are timing out intermittently, but as state grows they are timing out more and more often eventually killing the job. Size of the state is large with Minimum=10.2MB and Maximum=49GB (this one is accumulated due to prior

Re: Need guidance on a use case

2019-12-27 Thread Eva Eva
>>> >>> I'm not 100% sure if your use case can be solved with SQL. JOIN in SQL >>> always joins an incoming record with all previous arrived records. Maybe >>> Jark in CC has some idea? >>> >>> It might make sense to use the DataStream API inste

Need guidance on a use case

2019-12-18 Thread Eva Eva
Hi Team, I'm trying Flink for the first time and encountered an issue that I would like to discuss and understand if there is a way to achieve my use case with Flink. *Use case:* I need to perform unbounded stream joins on multiple data streams by listening to different Kafka topics. I have a sce