Some more input:
Right now, you can also use the `ProcessFunction` [1] available in Flink 1.2 to
simulate state TTL.
The `ProcessFunction` should allow you to keep device state and simulate the
online / offline detection by registering processing timers. In the `onTimer`
callback, you can emit
Hi Steve,
I’ll try to provide some input for the approaches you’re currently looking into
(I’ll comment on your email below):
* API based stop and restart of job … ugly.
Yes, indeed ;) I think this is absolutely not necessary.
* Use a co-map function with the rules alone stream and the events
Hi Dominik,
I would recommend implementing a `KeyedSerializationSchema`, and supply it to
the constructor when initializing your FlinkKafkaConsumer.
The `KeyedDeserializationSchema` exposes the metadata of the record such as
offset, partition, and key. In the schema, you can implement your own
I think my previous guess was wrong. From what I can tell, when Kryo tries to
copy the exception object, it does that by serializing and deserializing it.
For subclasses of RuntimeException, it doesn't know how to do it so it
delegates serialization to Java. However, it doesn't use a custom
Obj
This happened when running Flink with bin/run-local.sh I notice that there only
appears to be one Java process. The job manager and the task manager run in the
same JVM, right? I notice, however, that there are two blob store folders on
disk. Could the problem be caused by two different FlinkUse
Has anyone encountered this or know what might be causing it?
java.lang.RuntimeException: Could not forward element to next operator
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:394)
at
org.apache.flink.streaming.run
I’ve been reading the code/user goup/SO and haven’t really found a great answer
to this… so I thought I’d ask.
I have a UI that allows the user to edit rules which include specific criteria
for example trigger event if X many people present for over a minute.
I would like to have a flink job th
Hi Folks,
We are working on a Flink job to proccess a large amount of data coming in
from a Kafka stream.
We selected Flink because the data is sometimes out of order or late, and we
need to roll up the data into 30-minutes event time windows, after which we
are writing it back out to an s3 bucke
Hi,
Unfortunately I cannot find the option of using raw ConsumerRecord
instances when creating a Kafka data stream.
In general, I would like to use an instance of the mentioned type because our
use case requires certain metadata such as record offset and partition.
So far I’ve examined the so
Hi Yassine,
Thanks for reporting this. The problem you run into is due to start-local.sh
which we discourage in favour of start-cluster.sh that resembles real use case
better.
In your case, start-local.sh starts a job manager with an embedded task
manager but does not parse the task manager con
I spoke to one of the representatives in AWS EMR team last week. They
mentioned that they usually practice a 4 week cool down period. Hopefully
we will get Flink 1.2 in the next week.
Thanks,
Sandeep
On Mar 6, 2017 9:27 AM, "Chen Qin" wrote:
> EMR is a team within Amazon Web Services, to them,
EMR is a team within Amazon Web Services, to them, Flink is one of frameworks
they need to support. Haven't see SLA around release time lag, but seems common
practice better off keep a cool down period and let problem resolved before
onboard new versions.
Sent from my iPhone
> On Mar 6, 2017,
Does anyone have any insight as to how closely Amazon EMR releases will track
the Flink releases? For example EMR 5.3.1 on Feb 7 included Flink 1.1.4, from
Dec 21 ... about 1.5 month lag. With Flink accelerating to a timed release
schedule, I wonder how far behind EMR will track the official
Hi,
I am trying to achieve a stream-to-stream join with big windows and are
searching for a way to clean up state of old keys. I am already using a
RichCoProcessFunction
I found there is already an existing ticket
https://issues.apache.org/jira/browse/FLINK-3089
but I have doubts that a registr
I think I found the reason for what happened. The way I used the
QueryableStateClient is that I wrapped scala.concurrent.Future in a
FlinkFuture and then called FlinkFuture.thenAccept. It turns out
thenAccept doesn't
throw exceptions and when an exception happens (which likely happened once
I inrea
Hi Guys,
I am getting the same exception:
EMRFileSystem not Found
I am trying to read encrypted S3 file using Hadoop File System class.
(using Flink 1.2.0)
When I copy all the libs from /usr/share/aws/emrfs/lib and /usr/lib/hadoop
to Flink lib folder , it works.
However I see that all these lib
Hi all,
I set up a job with simple queryable state sink and tried to query it from
another job using the new Async I/O API. Everything worked as expected,
except when I tried to increase the parallelism of the querying job it
hanged.
As you can see in the attached image, when the parallism is 5 (e
Hi,
Are you running Zeppelin on a local machine?
I haven’t tried this before, but you could first try and check if port ‘6123’
is publicly accessible in the security group settings of the AWS EMR instances.
- Gordon
On March 3, 2017 at 10:21:41 AM, Meghashyam Sandeep V (vr1meghash...@gmail.co
Hi Dominik,
AFAIK, the local mode executions create a mini cluster within the JVM to run
the job.
Also, `MiniCluster` seems to be something FLIP-6 related, and since FLIP-6 is
still work
in progress, I’m not entirely sure if it is viable at the moment. Right now,
you should look
into using `Lo
Hi Bruno!
The Flink CEP library also seems like an option you can look into to see if it
can easily realize what you have in mind.
Basically, the pattern you are detecting is a timeout of 5 minutes after the
last event. Once that pattern is detected, you emit a “device offline” event
downstrea
Hello,
i believe the cassandra connector is not shading it's dependencies
properly. This didn't cause issues in the
past since flink used to have a dependency on codahale metrics as well.
Please open a JIRA for this issue.
Regards,
Chesnay
On 06.03.2017 11:32, Tarandeep Singh wrote:
Hi Robe
Hi Robert & Nico,
I am facing the same problem (java.lang.NoClassDefFoundError:
com/codahale/metrics/Metric)
Can you help me identify shading issue in pom.xml file.
My pom.xml content-
-
http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org
22 matches
Mail list logo
