Hi All,
We are using Flink Kubernetes Operator on our production. We have 3k+ jobs
in standalone mode. But after 2.5k jobs operator getting slow. Now when we
submit a job it takes 10+ minutes to the job runs. Does anyone use similar
scale or more job ?
Now we run as a single pod. Does operator su
Hi!
It’s currently not possible to run the operator in parallel by simply
adding more replicas. However there are different things you can do to
scale both vertically and horizontally.
First of all you can run multiple operators each watching different set of
namespaces to partition the load.
Th
Thanks Tamir for the information. According to the latest comment of the
task FLINK-24998, this bug should be gone while using the latest JDK 17. I
was wondering whether it means that there are no more issues to stop us
releasing a major Flink version to support Java 17? Did I miss something?
Best
Hi Giannis,
I'm curious, what tool did you use for this analysis (what the screenshot
shows)? Is it something custom?
Thank you.
On Wed, Apr 26, 2023 at 10:38 PM Giannis Polyzos
wrote:
> This is really helpful,
>
> Thanks
>
> On Thu, Apr 27, 2023 at 5:46 AM Yanfei Lei wrote:
>
>> Hi Giannis,
Correct, its some custom code i put together to investigate what gets
written in rocksdb
On Thu, Apr 27, 2023 at 6:06 PM Yaroslav Tkachenko
wrote:
> Hi Giannis,
>
> I'm curious, what tool did you use for this analysis (what the screenshot
> shows)? Is it something custom?
>
> Thank you.
>
> On W
Got it! Any chance you can open-source some of that? I think it can be
extremely useful for the community.
Thank you.
On Thu, Apr 27, 2023 at 8:08 AM Giannis Polyzos
wrote:
> Correct, its some custom code i put together to investigate what gets
> written in rocksdb
>
> On Thu, Apr 27, 2023 at 6
Will definitely do as it's going to be part of a wider Flink course / book
(haven't decided yet on the format) Im putting together.
but I can share before that If you want
On Thu, Apr 27, 2023 at 6:11 PM Yaroslav Tkachenko
wrote:
> Got it! Any chance you can open-source some of that? I think it
Hello!
As a newcomer to Flink and Kubernetes, I am seeking detailed instructions
to help me properly configure and deploy this Flink example (
https://github.com/apache/flink-kubernetes-operator/tree/main/examples/autoscaling)
on a Minikube environment. Can you give me specific configurations and
Scala 2.12.7 doesn't compile on Java 17, see
https://issues.apache.org/jira/browse/FLINK-25000.
On Thu, Apr 27, 2023 at 3:11 PM Jing Ge wrote:
> Thanks Tamir for the information. According to the latest comment of the
> task FLINK-24998, this bug should be gone while using the latest JDK 17. I
>
Is the intention to bump the Flink major version and only support Java 17+?
If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser
wrote:
> Scala 2.12.7 doesn't compile on Java 17, see
> https://issues.apache.org/jira/browse/FLINK-25000
Hi, rania,
I think the quick start document[1] is helpful for you. Other information
could be found in its documents[2].
Best,
Hang
[1]
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/
[2] https://nightlies.apache.org/flink/fl
Thank you for all your responses! I think Gyula is right, simply do a MAX -
some_offset is not ideal as it can make the standby TM useless.
It is difficult for the scheduler to determine whether a pod has been lost
or scaled down when we enable autoscaling, which affects its decision to
utilize sta
You could also check out the Autoscaler logic in the Flink Kubernetes
Operator (
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/
)
On the current main and in the upcoming 1.5.0 release the mechanism is
pretty nice and solid :)
It works with t
13 matches
Mail list logo
