Hi Till,
Thanks for your reply.
>> My suggestion would be to split the different patterns up and run them
with in different jobs.
I'm not able to understand how splitting up the jobs based on patterns
would be more efficient than based on the key. The total number of
operators would still be the
Hi Shailesh,
I fear that given your job topology, it is not that surprising that things
break. The problem is that you might have M x N CEP operators concurrently
active. This means that they have to keep their state in memory. Given 3.5
GB isn't that much if you have more than 300 CEP NFAs runnin
Hi Till,
When I'm submitting one big job, both JM and TM (sometimes just JM) are
crashing at the time of initialization itself (i.e. not all operators
switch to RUNNING) with OOM. The number of threads on TM go to almost 1000.
But when I'm submitting multiple jobs, job submission is completed. Bu
Hi Shailesh,
my question would be where do you see the OOM happening? Does it happen on
the JM or the TM.
The memory requirements for each operator strongly depend on the operator
and it is hard to give a general formula for that. It mostly depends on the
user function. Flink itself should not ne
Actually, there are too many hyperparameters to experiment with, that is
why I'm trying to understand if there is any particular way in which a
cluster could be benchmarked.
Another strange behaviour I am observing is: Delaying the operator creation
(by distributing the operators across jobs, and
Hi,
You could definitely try to find formula for heap size, but isnt's it
easier just to try out different memory settings and see which works best
for you?
Thanks,
Pawel
17 lut 2018 12:26 "Shailesh Jain" napisał(a):
Oops, hit send by mistake.
In the configuration section, it is mentioned tha
Oops, hit send by mistake.
In the configuration section, it is mentioned that for "many operators"
heap size should be increased.
"JVM heap size (in megabytes) for the JobManager. You may have to increase
the heap size for the JobManager if you are running very large applications
(with many opera
