Flink on MapR

Hi everyone, short question about running Flink on MapR. Is it still necessary to compile with MapR libraries or it is enough to take binary distribution and put MapR jars in classpath? Would really appreciate feedback from someone running such combination in production, especially if MapR is sec

Re: Set partition number of Flink DataSet

Hi Ken, Do you mean that I can create a batch sink which writes to N files? That sounds viable, but since our data size is huge (billions of records & thousands of files), the performance may be unacceptable. I will check Blink and give it a try anyway. Thank you, Qi > On Mar 12, 2019, at 11:

Re: Understanding timestamp and watermark assignment errors

Hi Andrew, generally, this looks like a concurrency problem. Are you using asynchronous checkpointing? If so, could you check if this issue also occurs with synchronous checkpointing. There have been reports recently, that there might be a problem with some Kryo types. Can you set the logging le

Partitions and the number of cores/executors

Hi, I am working with Gelly graph library but I think the question is applicable in general. I just want to confirm if a single data partition in Flink is executed by only a single executor/core? i.e. multiple executors can't be utilized to process a single partition in parallel. So, if I need

Challenges using Flink REST API

Hey all! I'm looking for some advice on the following; I'm working on an abstraction on top of Apache Flink to 'pipeline' Flink applications using Kafka. For deployment this means that all these Flink jobs are embedded into one jar and each job is started using an program argument (e.g. "--stage '

Re: Challenges using Flink REST API

You should get the full stacktrace if you upgrade to 1.7.2 . On 13.03.2019 09:55, Wouter Zorgdrager wrote: Hey all! I'm looking for some advice on the following; I'm working on an abstraction on top of Apache Flink to 'pipeline' Flink applications using Kafka. For deployment this means that a

Custom Partitioner and Graph Algorithms

Hi, I am observing a behaviour in the task statistics that I don't fully understand. Essentially I have create a partitioner that assigns all the edges to a single partition. I see imbalance (in terms of records sent/received) in the task statistics of different instances of the same operator

Re: Challenges using Flink REST API

Hi Chesnay, Unfortunately this is not true when I run the Flink 1.7.2 docker images. The response is still: { "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] } Regards, Wouter Zorgdrager Op wo 13 mrt. 2019 om 10:42 schr

Re: Challenges using Flink REST API

Can you give me the stacktrace that is logged in the JobManager logs? On 13.03.2019 10:57, Wouter Zorgdrager wrote: Hi Chesnay, Unfortunately this is not true when I run the Flink 1.7.2 docker images. The response is still: { "errors": [ "org.apache.flink.client.program.ProgramInvocationE

Re: Challenges using Flink REST API

Hey Chesnay, Actually I was mistaken by stating that in the JobManager logs I got the full stacktrace because I actually got the following there: 2019-03-13 11:55:13,906 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler- Exception occurred in REST handler: org.apache.flink.clien

Will state TTL support event time cleanup in 1.8?

Do improvements introduced in https://issues.apache.org/jira/browse/FLINK-10471 add support for event time TTL?

Re: Understanding timestamp and watermark assignment errors

Hi, I think this looks like the same problem as in this issue: https://issues.apache.org/jira/browse/FLINK-11420 Best, Stefan > On 13. Mar 2019, at 09:41, Konstantin Knauf wrote: > > Hi Andrew, > > generally, this looks like a concurrenc

Re: Will state TTL support event time cleanup in 1.8?

TTL based on event time is not part or 1.8, but likely to be part of 1.9. > On 13. Mar 2019, at 13:17, Sergei Poganshev wrote: > > Do improvements introduced in > https://issues.apache.org/jira/browse/FLINK-10471 > add support for event > ti

Re: Migrating Existing TTL State to 1.8

Just wondering if anyone has any insights into the new TTL state cleanup feature mentioned below. Thanks, — Ning > On Mar 11, 2019, at 1:15 PM, Ning Shi wrote: > > It's exciting to see TTL state cleanup feature in 1.8. I have a question > regarding the migration of existing TTL state to the

Re: Challenges using Flink REST API

My bad, I was looking at the wrong code path. The linked issue isn't helpful, as it only slightly extends the exception message. You cannot get the stacktrace in 1.7.X nor in the current RC for 1.8.0 . I've filed https://issues.apache.org/jira/browse/FLINK-11902 to change this. The 1.8.0 RC j

Flink to MQ connector with checkpoint support for exctly once semantics

Hi, Our Flink jobs need to read messages from IBM MQ and write messages into IBM. Just wonder if there are already some MQ connectors with two phase committee sink Function or Checkpoint Listener and checkpoint function implemented to support the exactly once semantics. Many thanks in advance.

Batch jobs stalling after initial progress

Hi, I'm running flink batch jobs on EMR 5.21, and I'm seeing many (>50%) jobs stall and make no progress after some initial period. I've seen the behaviour earlier (5.17), but not nearly as much as now. The job is a fairly simple enrichment job, loading an avro metadata file, creating several dat

Re: K8s job cluster and cancel and resume from a save point ?

BTW, does 1.8 also solve the issue where we can cancel with a save point. That too is broken in 1.7.2 curl --header "Content-Type: application/json" --request POST --data '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":*true*}' https://*/jobs/00

Re: Partitions and the number of cores/executors

Hi, Your assumption is right. Parallel processing is based in splitting inputs and each split is only processed by one task instance at a time. Best, Stefan > On 13. Mar 2019, at 09:52, mbilalce@gmail.com wrote: > > Hi, > > I am working with Gelly graph library but I think the question is

Re: Migrating Existing TTL State to 1.8

Hi, If you are worried about old state, you can combine the compaction filter based TTL with other cleanup strategies (see docs). For example, setting `cleanupFullSnapshot` when you take a savepoint it will be cleared of any expired state and you can then use it to bring it into Flink 1.8. Bes

Scala API support plans

Hi guys, what are the plans for scala-api support? Asking about it because I’ve recently found that scala-api used to catch up with java one. Thanks!

Re: Backoff strategies for async IO functions?

Thanks for raising the concern @shuyi and the explanation @konstantin. Upon glancing on the Flink document, it seems like user have full control on the timeout behavior [1]. But unlike AsyncWaitOperator, it is not straightforward to access the internal state of the operator to, for example, put th

Re: [DISCUSS] Create a Flink ecosystem website

@Bowen: I agree. Confluent Hub looks nicer, but it is on their company website. I guess the likelihood that they give out code from their company website is fairly low. @Nils: Beam's page is similar to our Ecosystem page, which we'll reactivate as part of this PR: https://github.com/apache/flink-we

Re: Set partition number of Flink DataSet

Hi Qi, > On Mar 13, 2019, at 1:26 AM, qi luo wrote: > > Hi Ken, > > Do you mean that I can create a batch sink which writes to N files? Correct. > That sounds viable, but since our data size is huge (billions of records & > thousands of files), the performance may be unacceptable. The main

Re: Batch jobs stalling after initial progress

Hi Marko, Some things that have caused my jobs to run very slowly (though not completely stall) 1. Cross-joins generating huge result sets. 2. Joins causing very large spills to disk. 3. Slow external API access With streaming, iterations can cause stalls, but I don’t think that’s true for

Re: Conflicting Cassandra versions while using flink-connector-cassandra

Just created a PR trying to address this issue: https://github.com/apache/flink/pull/7980 what do you think? Em ter, 12 de mar de 2019 às 23:23, Gustavo Momenté < momente.gust...@gmail.com> escreveu: > Can I shade `flink-connector-cassandra` version? And if so do you know why > it isn't shaded by

Re: How to join stream and dimension data in Flink?

Hi Henry, These are good questions! I would rather not to add the temporal and lateral prefix in front of the join. The temporal table is a concept orthogonal to join. We should say join a temporal table or join a Lateral table. 1. You can of course use stream-stream join. Introducing the temporal

Re: Set partition number of Flink DataSet

Hi Ken, Agree. I will try partitonBy() to reducer the number of parallel sinks, and may also try sortPartition() so each sink could write files one by one. Looking forward to your solution. :) Thanks, Qi > On Mar 14, 2019, at 2:54 AM, Ken Krugler wrote: > > Hi Qi, > >> On Mar 13, 2019, at 1

What should I take care if I enable object reuse

Hi Community, I saw from the document that we need to be careful about enable the object reuse feature. So which part should I check to avoid any issues? Can any one help to summarize? Thank you. // *enableObjectReuse() / disableObjectReuse()* By default, objects are not reused in Flink. Enab