Re: Missing MapState when Timer fires after restored state

Hi Juho, As Sihua said, this shouldn't happen and indicates a bug. Did you only encounter this once or can you easily reproduce the problem? Best, Aljoscha > On 15. May 2018, at 05:57, sihua zhou wrote: > > Hi Juho, > in fact, from your code I can't see any possible that the MapState could be

Re: Missing MapState when Timer fires after restored state

Hi Juho, in fact, from your code I can't see any possible that the MapState could be inconsistency with the timer, it's looks like a bug to me, because once the checkpoint's complete and you haven't query the state in a customer thread async, then the result of the checkpoint should be consisten

Leader Retrieval Timeout with HA Job Manager

Hi, I am using the 1.4.2 release on ubuntu and attempting to make use of an HA Job Manager, but unfortunately using HA functionality prevents job submission with the following error: java.lang.RuntimeException: Failed to retrieve JobManager address     at org.apache.flink.client.program.Cl

minPauseBetweenCheckpoints for failed checkpoints

Hello! I have a question regarding checkpointing parameter minPauseBetweenCheckpoints that is the minimal pause between checkpointing attempts. I’ve noticed the following (strange) behavior in Flink. I set the following parameters for a sample Flink job: Checkpointing Mode = Exactly Once Interv

Re: Best way to clean-up states in memory

Thanks Fabian, Kostas, Here is what I had in the Trigger - idea is to run bitwise OR until a threshold is reached or a timeout is reached (nothing too fancy here). Let me know what you guys think. Like I said, I moved this logic to Process Function and I haven't seen the same issue I was with t

Flink does not read from some Kafka Partitions

Hello, My team ran into some behavior we did not expect when we tried to get an existing Flink app to read from a re-sized Kafka. Here are the highlights: - We are using the FlinkKafkaConsumer010. - We re-partitioned (added partitions to) an existing topic that our Flink app reads so that it th

Akka heartbeat configurations

Hi, We are running into issues where GC pause will result into Taskmanagers being marked dead incorrectly. Flink documentation documents some knobs of Akka configurations to play aroun

Re: Checkpoint is not triggering as per configuration

Thanks for the reply Piotr. Which jira ticket were you refer to? We were trying to use the same code for normal stream process to process very old historical backfill data. The problem for me right now is that, backfill x years of data will be very slow. And I cannot have any checkpoint during the

Re: Missing MapState when Timer fires after restored state

Hi Juho, You are right, there's no transactional guarantee on timers and state in processElement(). They may end up with inconsistency if your job was cancelled in the middle of processing an element. To avoid the situation, the best programming practice is to always check if the state you're try

AvroInputFormat Serialisation Issue

Hi all - sorry this seems like a silly question, but I can't figure it out. I'm using an AvroInputFormat in order to read an Avro file like this: val textInputFormat = new AvroInputFormat[GenericRecord](infile, classOf[GenericRecord]) val lines = env.readFile(textInputFormat, path) This works fi

Re: Consumed input splits

Anyone on this? Am I the only one that find this UI feature useful? :( On Thu, Apr 26, 2018 at 2:54 PM, Flavio Pompermaier wrote: > Hi to all, > is there a way to see from the Flink UI the number of procesed splits o a > source? > For example, I'm reading data from a (batch) JDBC input and Flink

Re: Consolidated log for a job?

Hi Alexey, I know that Kibana(https://en.wikipedia.org/wiki/Kibana) can show logs from different servers at one screen. May be this is what you are looking for Alex On Mon, May 14, 2018 at 5:17 PM NEKRASSOV, ALEXEI wrote: > Is there a way to see logs from multiple Task Managers **all in one pl

Re: How to set fix JobId for my application.

Thanks for suggestion now using Kafka for information sharing between apps. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Consolidated log for a job?

Is there a way to see logs from multiple Task Managers *all in one place* (for a running or a completed job)? Or I need to check logs on each Task Manager individually? Thanks, Alex Nekrassov

Re: Better way to clean up state when connect

Hi Xingcan, Thanks for your response, to give your more background about my use case, I have Stream B with some split test name, and Stream A will be the actual test. I want to have Stream A connect to Stream B to figure out whether this test is still active or not. I am not sure this is the right

Re: A strange exception while consumption using a multi topic Kafka Connector

Yep, Thanks. We have a set up where topics are constantly being added on a kakfa-to-hdfs pipeline. On Sun, May 13, 2018 at 11:58 AM, Ted Yu wrote: > FLINK-9349 was logged. > > FYI > > On Sat, May 12, 2018 at 7:52 AM, Ted Yu wrote: > >> I took a look at ./flink-connectors/flink-co >> nnector-kaf

Async Source Function in Flink

Hello everyone, just wanted to ask a quick question: I have to retrieve data from 2 web services via REST calls, use them as sources and push these data to Kafka. So far, I implemented a SourceFunction which deals with making the calls with the respective clients. Now, the function does use, for

Re: Taskmanager JVM crash

No, that problem I mentioned does not affect batch jobs. Must be something different then, but unfortunately the dump looks not very helpful to me because of the „error occurred during error reporting (printing native stack)“. > Am 14.05.2018 um 14:26 schrieb Flavio Pompermaier : > > My job is

Re: Taskmanager JVM crash

My job is a batch one, not a streaming job. Is it possible that the cause is the one you mentioned? On Mon, 14 May 2018, 14:23 Stefan Richter, wrote: > Hi, > > that looks like a known issue where Flink did not wait for the shutdown of > the timer service before disposing state backends. This is

Re: Taskmanager JVM crash

Hi, that looks like a known issue where Flink did not wait for the shutdown of the timer service before disposing state backends. This is problem fixed in the >= 1.4 branches. Best, Stefan > Am 14.05.2018 um 14:12 schrieb Flavio Pompermaier : > > Hi to all, > I have a Flink 1.3.1 job that ru

Taskmanager JVM crash

Hi to all, I have a Flink 1.3.1 job that runs multiple times. Everything goes well for some time (e.g. 10 jobs). Then, one or more TMs suddently die. In the .out file I find something like this: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7f

Missing MapState when Timer fires after restored state

We have a Flink streaming job (1.5-SNAPSHOT) that uses timers to clear old state. After restoring state from a checkpoint, it seems like a timer had been restored, but not the data that was expected to be in a related MapState if such timer has been added. The way I see this is that there's a bug,

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer01

Getting below error. *Command: * ./bin/flink run ~/workspace/thanos/thanos-stream/target/thanos-stream.jar --port 9000 It works well from IDE and java -jar command. Cluster configuration: Standalone cluster with JobManager at localhost/ 127.0.0.1:6123 Using address localhost:6123 to connect to

Re: Best way to clean-up states in memory

Hi Ashish, Did you use per-window state (also called partitioned state) in your Trigger? If yes, you need to make sure that it is completely removed in the clear() method because processing time timers won't fire once a window was purged. So you cannot (fully) rely on timers to clean up per-window

Re: Late data before window end is even close

Thanks for correcting me Piotr. I didn't look close enough at the code. With the presently implemented logic, a record should not be emitted to a side output if its window wasn't closed yet. 2018-05-11 14:13 GMT+02:00 Piotr Nowojski : > Generally speaking best practise is always to simplify your

Flink FSStateBackend Checkpointing Buffer size

Hi, I am trying to use Gluster File System as my FileSystem backed by RocksDB as state backend. I can see from FsCheckpointStateOutputStream that the  DEFAULT_WRITE_BUFFER_SIZE = 4096. Is the buffer size configurable in any way? Any idea about the checkpointing performance with default buffer siz

Re: How to broadcast messages to all task manager instances in cluster?

Hi, Thanks for the clarification. This might be though. Generally speaking having such static configuration shared across multiple operators/functions can pose lots of different problems including synchronisation, fault tolerance etc. To be honest you should treat such thing almost like an ext

Re: Batch writing from Flink streaming job

Hi, Avro provides schema for data and can be used to serialize individual records in a binary format. It does not compress the data (although this can be put on top) but is more space efficient due to the binary serialization. I think you can implement a Writer for the BucketingSink that writes r

Re: Default zookeeper

No, there is no difference if the version in your distro is part of the ZooKeeper 3.4.x series. The script is there for convenience during local testing/dev. – Ufuk On Sun, May 13, 2018 at 3:49 PM, miki haiat wrote: > When downloading the the flink source in order to run it local thire is a > z

Re: Best way to clean-up states in memory

Hi Ashish, It would be helpful to share the code of your custom trigger for the first case. Without that, we cannot tell what state you create and how/when you update/clear it. Cheers, Kostas > On May 14, 2018, at 1:04 AM, ashish pok wrote: > > Hi Till, > > Thanks for getting back. I am sure