Hi Flavio,
We have implemented our own flink operator, the operator will start a flink
job cluster (the application jar is already packaged together with flink in
the docker image). I believe Google's flink operator will start a session
cluster, and user can submit the flink job via REST. Not look
Sorry I wanted to mention https://github.com/lyft/flinkk8soperator (I don't
know which one of the 2 is better)
On Wed, Mar 11, 2020 at 10:19 AM Flavio Pompermaier
wrote:
> Have you tried to use existing operators such as
> https://github.com/GoogleCloudPlatform/flink-on-k8s-operator or
> https:/
Have you tried to use existing operators such as
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator or
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator?
On Wed, Mar 11, 2020 at 4:46 AM Xintong Song wrote:
> Hi Eleanore,
>
> That does't sound like a scaling issue. It's proba
Hi Eleanore,
That does't sound like a scaling issue. It's probably a data skew, that the
data volume on some of the keys are significantly higher than others. I'm
not familiar with this area though, and have copied Jark for you, who is
one of the community experts in this area.
Thank you~
Xinton
_Hi Xintong,
Thanks for the prompt reply! To answer your question:
- Which Flink version are you using?
v1.8.2
- Is this skew observed only after a scaling-up? What happens if the
parallelism is initially set to the scaled-up value?
I also tried this, it
Hi Eleanore,
I have a few more questions regarding your issue.
- Which Flink version are you using?
- Is this skew observed only after a scaling-up? What happens if the
parallelism is initially set to the scaled-up value?
- Keeping the job running a while after the scale-up, does the
Hi Experts,
I have my flink application running on Kubernetes, initially with 1 Job
Manager, and 2 Task Managers.
Then we have the custom operator that watches for the CRD, when the CRD
replicas changed, it will patch the Flink Job Manager deployment
parallelism and max parallelism according to th
