Hi All,
I see the rescale api allow us to somehow redistribute element locally, but is
it possible to make the upstream operator distributed evenly on task managers?
For example I have 10 task managers each with 10 slots. The application reads
data from Kafka topic with 20 partitions, then resca
l see
> none heap increases for every restart until you get an OOM.
>
> I let you know if/when I know how to handle the problem.
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
>> Den 3. sep. 2018 kl. 10.08 skrev 祁明良 :
>>
>> Hi All,
>>
Hi All,
We are running flink(version 1.5.2) on k8s with rocksdb backend.
Each time when the job is cancelled and restarted, we face OOMKilled problem
from the container.
In our case, we only assign 15% of container memory to JVM and leave others to
rocksdb.
To us, it looks like memory used by ro
Hi All,
When using FlinkKafkaProducer09 (Flink version 1.4.2), I’m facing an Kafka
batch expired error when checkpoint starts. The error log is attached below.
Here is what I have investigated:
1. The error only and always occurs when checkpoint starts.
2. The error seems not related to flushOnC
Hi all,
I’m currently using the keyed process function, I see there’s serialization
happening when I collect the object / update the object to rocksdb. For me the
performance of serialization seems to be the bottleneck.
By default, POJO serializer is used, and the timecost of collect / update to
ckpressure is a hint for that because TMs multiplex the connection of all
tasks.
Regarding the memory configuration question, I am not sure if there is a way to
override the JVM heap size on YARN. Maybe others can answer this question.
Best,
Fabian
2018-08-12 18:36 GMT+02:00 祁明良
mailto:m...@
Hi all,
We are trying to build our flink cluster on k8s, but there seems to be not
enough materials about details.
Firstly, we learned how to build a standalone cluster on k8s and run all the
jobs inside it.
The question is like are we building a cluster for all the jobs or one job per
clust
Hi all,
This is mingliang, I got a problem with rocksdb backend.
I'm currently using a 15min SessionWindow which also fires every 10s, there's
no pre-aggregation, so the input of WindowFunction would be the whole Iterator
of input object.
For window operator, I assume this collection is also
