IOException: Size of the state is larger than the maximum permitted memory-backed state

2018-05-01 Thread syed
Hi; I am facing the following IO exception while running a simple wordcount application. *IOException: Size of the state is larger than the maximum permitted memory-backed state. Size=34521369 , maxSize=5242880 . Consider using a different state backend, like the File System State backend.* I hav

FLIP-6 Docker / Kubernetes

2018-05-01 Thread anaray
Hi, I was reading *Docker/Kubernetes* section of FLIP-6 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077). I have question regarding point 1 ie "A job-specific image with the job libraries, configured to launch the job (with a JobManager)" How does job submission happens

How to log uncaught exceptions during job execution

2018-05-01 Thread Tzanko Matev
Hello, I am trying to attach Sentry to our Flink cluster to track job execution. Sentry acts as a logger which captures messages and sends them to a central server. By default it captures all messages with level WARN or higher. To get Sentry to catch all problems, I need to write a WARN or ERROR

Windows aligned with EDT ( with day light saving )

2018-05-01 Thread Vishal Santoshi
How do I align a Window with EDT with day light saving correction ? The offset takes a hardcoded value. I need 6 hour windows aligned to 00, 12 , 18 and so on but on EDT.

Re: KafkaProducer with generic (Avro) serialization schema

2018-05-01 Thread Wouter Zorgdrager
So, I'm still struggling with this issue. I dived a bit more into the problem and I'm pretty sure that the problem is that I have to (implicitly) pass the SchemaFor and RecordTo classes to my serialization schema (otherwise I can't make it generic). However those class aren't serializable, but of c

ApacheCon North America 2018 schedule is now live.

2018-05-01 Thread Rich Bowen
Dear Apache Enthusiast, We are pleased to announce our schedule for ApacheCon North America 2018. ApacheCon will be held September 23-27 at the Montreal Marriott Chateau Champlain in Montreal, Canada. Registration is open! The early bird rate of $575 lasts until July 21, at which time it goe

ConnectedIterativeStreams and processing state 1.4.2

2018-05-01 Thread Lasse Nedergaard
Hi. I have a case where I have a input stream that I want to enrich with external data. I want to cache some of the external lookup data to improve the overall performances. To update my cache (a CoProcessFunction) I would use iteration to send the external enriched information back to the cache a