A bit of extra information on the example where I posted the link:
The example checks whether two events follow each other within a certain
time:
- The first event in the example is called "compute.instance.create.start"
(in your case, it would be the event that an order was placed)
- The seco
Hi Mihail!
Do I understand you correctly that the use case is to raise an alarm if an
order has not been processed within a certain time period (certain number
of days) ?
If that is the case, the use case is actually perfect for a special form of
session windows that monitor such timeouts. I have
Hi Mihail,
not sure if I correctly got your requirements, but you can define windows
on a keyed stream. This basically means that you partition the stream, for
example by order-id, and compute windows over the keyed stream. This will
create one (or more, depending on the window type) window for ea
Hi Gyula, Hi Stephan,
thank you for your replies.
We need a state which grows indefinitely for the following use case. An
event is created when a customer places an order. Another event is created
when the order is sent. These events typically occur within days. We need
to catch the cases when th
Mihail!
The Flink windows are currently in-memory only. There are plans to relax
that, but for the time being, having enough memory in the cluster is
important.
@Gyula: I think window state is currently also limited when using the
SqlStateBackend, by the size of a row in the database (because win
Hi,
I am working on a use case that involves storing state for billions of
keys. For this we use a MySql state backend that will write each key-value
state to MySql server so it will only hold a limited set of key-value pairs
on heap while maintaining the processing guarantees.
This will keep our
Hi Aljoscha,
we have no upper bound for the number of expected keys. The max size for an
element is 1 KB.
There are 2 Maps, a KeyBy, a timeWindow, a Reduce and a writeAsText
operators in the job. In the first Map we parse the contained JSON object
in each element and forward it as a Flink Tuple.
Hi Mihail,
could you please give some information about the number of keys that you are
expecting in the data and how big the elements are that you are processing in
the window.
Also, are there any other operations that could be taxing on Memory. I think
the different exception you see for 500M
Yes, with the "start-cluster-streaming.sh" script.
If the TaskManager gets 5GB of heap it manages to process ~100 million
messages and then throws the above OOM.
If it gets only 500MB it manages to process ~8 million and a somewhat
misleading exception is thrown:
12/01/2015 19:14:07Source: Cus
Its good news that the issue has been resolved.
Regarding the OOM, did you start Flink in the streaming mode?
On Wed, Dec 2, 2015 at 10:18 AM, Vieru, Mihail
wrote:
> Thank you, Robert! The issue with Kafka is now solved with the
> 0.10-SNAPSHOT dependency.
>
> We have run into an OutOfMemory ex
Thank you, Robert! The issue with Kafka is now solved with the
0.10-SNAPSHOT dependency.
We have run into an OutOfMemory exception though, which appears to be
related to the state. As my colleague, Javier Lopez, mentioned in a
previous thread, state handling is crucial for our use case. And as the
Thanks! I've linked the issue in JIRA.
On Tue, Dec 1, 2015 at 5:39 PM, Robert Metzger wrote:
> I think its this one https://issues.apache.org/jira/browse/KAFKA-824
>
> On Tue, Dec 1, 2015 at 5:37 PM, Maximilian Michels wrote:
>>
>> I know this has been fixed already but, out of curiosity, could
I think its this one https://issues.apache.org/jira/browse/KAFKA-824
On Tue, Dec 1, 2015 at 5:37 PM, Maximilian Michels wrote:
> I know this has been fixed already but, out of curiosity, could you
> point me to the Kafka JIRA issue for this
> bug? From the Flink issue it looks like this is a Zoo
I know this has been fixed already but, out of curiosity, could you
point me to the Kafka JIRA issue for this
bug? From the Flink issue it looks like this is a Zookeeper version mismatch.
On Tue, Dec 1, 2015 at 5:16 PM, Robert Metzger wrote:
> Hi Gyula,
>
> no, I didn't ;) We still deploy 0.10-SN
Hi Gyula,
no, I didn't ;) We still deploy 0.10-SNAPSHOT versions from the
"release-0.10" branch to Apache's maven snapshot repository.
I don't think Mihail's code will run when he's compiling it against
1.0-SNAPSHOT.
On Tue, Dec 1, 2015 at 5:13 PM, Gyula Fóra wrote:
> Hi,
>
> I think Robert
Hi,
I think Robert meant to write setting the connector dependency to
1.0-SNAPSHOT.
Cheers,
Gyula
Robert Metzger ezt írta (időpont: 2015. dec. 1., K,
17:10):
> Hi Mihail,
>
> the issue is actually a bug in Kafka. We have a JIRA in Flink for this as
> well: https://issues.apache.org/jira/browse
Hi Mihail,
the issue is actually a bug in Kafka. We have a JIRA in Flink for this as
well: https://issues.apache.org/jira/browse/FLINK-3067
Sadly, we haven't released a fix for it yet. Flink 0.10.2 will contain a
fix.
Since the kafka connector is not contained in the flink binary, you can
just s
Hi,
we get the following NullPointerException after ~50 minutes when running a
streaming job with windowing and state that reads data from Kafka and
writes the result to local FS.
There are around 170 million messages to be processed, Flink 0.10.1 stops
at ~8 million.
Flink runs locally, started w
18 matches
Mail list logo
