e between using StateFun's
metrics and using normal flink metrics. Once you have configured the
Prometheus metric reporter following the doc
<https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/metrics/>,
maybe you can check the collected metrics to see if there are some a
tor jobs started by flink k8s operator. Maybe it
can be helpful.
Best,
Biao Geng
Oliver Schmied 于2024年5月19日周日 04:21写道：
> Dear Apache Flink Community,
>
> I am currently trying to monitor an Apache Flink cluster deployed on
> Kubernetes using Prometheus and Grafana. Despite fo
once
they are considered as unnecessary to keep.
Best,
Biao Geng
Anton Sidorov 于2024年5月16日周四 16:12写道：
> Hello!
>
> I have a Flink Job with CEP pattern.
>
> Pattern example:
>
> // Strict Contiguity
> // a b+ c d e
> Pattern.begin("a", Af
fer-OutOfMemoryError-when-using-Kafka-Connector-in-Flink>
.
Best,
Biao Geng
John Smith 于2024年5月16日周四 09:01写道：
> I deployed a new cluster, same version as my old cluster(1.14.4 ), only
> difference using Java 11 and it seems after a week of usage the below
> exception happens.
>
>
Typically it can help
you locate the over spawned threads.
As for the heap dump issue, it looks like a by-design behavior of the JDK.
See https://bugs.openjdk.org/browse/JDK-6784422 .
Best,
Biao Geng
Madan D via user 于2024年5月15日周三 09:43写道：
> Hello Team,
>
> Good morning!
> We ha
Hi Vararu,
Does this document meet your requirements?
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#logging
Best,
Biao Geng
Vararu, Vadim 于2024年5月14日周二 01:39写道：
> Hi,
>
>
>
> Trying to configure logger
Hi Sachin,
Your setting looks fine to me. If you want to verify that, one way is to
set the log level to 'trace' and check if logs like 'Ignoring metric {}.{}
for reporter #{} due to filter rules.' is printed.
Best,
Biao Geng
Sachin Mittal 于2024年5月12日周日 02:51写道：
>
ERRUPTION_NOTICE_DURATION
The method name SPOT_INSTANCE_INTERRUPTION_NOTICE_DURATION does not exist
in flink repo. Maybe it is better to contact the AWS expert to find the
cause of this problem.
Best,
Biao Geng
Phil Stavridis 于2024年5月8日周三 17:51写道：
> I am running a Flink job locally using
>
> python -m job.py
>
)"
will be executed and the following is a no-op. I think that's why you see
only the first job is submitted. In the JM log, you may see messages like
`Job 4eadb4b025efa5dc2a6ef2199af6ca2b was recovered successfully.` which
implies the following executeAsync() does not take effect.
Best,
B
ute().collect().next();
```
But I failed to reproduce the exception you attached :(
Best,
Biao Geng
Eduard Skhisov via user 于2024年5月1日周三 05:09写道：
> Hello,
>
> I am trying to use CloseableIterator, but next() operation reliably
> generates the following error:
>
>
>
Hi Abhi,
If your case can be reproduced steadily, have your ever tried to get
the thread dump of the TM which the problematic operator resides in?
Maybe we can get more clues with the thread dump to see where the
operator is getting stuck.
Best,
Biao Geng
Abhi Sagar Khatri via user 于2024年4月30日
Hi there,
Would you mind sharing the whole JM/TM log? It looks like the error log in
the previous email is not the root cause.
Best,
Biao Geng
ou...@139.com 于2024年4月29日周一 16:07写道：
> Hi all:
>When I ran flink sql datagen source and wrote to jdbc, checkpoint kept
> failing
-kubernetes-operator/quick-start/
Best,
Biao Geng
Hemi Grs 于2024年4月26日周五 19:03写道：
> Hi Biao,
>
> Thanks for your reply, fortunately the problem is solved.
> All I did was changed the bind-host to 0.0.0.0 (previously it was set to
> the server's IP).
>
> I don't kn
Hi Lee,
A quick question: what version of flink are you using for
testing execution.state-recovery.path?
It looks like this config is only supported in flink 1.20
<https://issues.apache.org/jira/browse/FLINK-34454> which is not released
yet.
Best,
Biao Geng
Lee, Keith 于2024年4月26日周五 0
Hi Hemi,
How do you start your flink cluster? Are you using standalone cluster or
using k8s/yarn as resource providers?
Also, it would be very helpful if you can share the full jobmanager log.
Best,
Biao Geng
Hemi Grs 于2024年4月18日周四 15:43写道：
> Hello,
>
> I have several versions of Flin
cords. Maybe they
are not what you are looking for.
Best,
Biao Geng
Jacob Rollings 于2024年4月18日周四 11:06写道：
> Hello,
>
> I have a use case where I need to do a cache file deletion after a
> successful sunk operation(writing to db). My Flink pipeline is built using
> Java. I am con
. https://issues.apache.org/jira/browse/FLINK-27392 This jira supports
defining window among adjacent sub-pattern.
3. https://issues.apache.org/jira/browse/FLINK-26941 This jira enables
Pattern to end with notFollowedBy with window
Best,
Biao Geng
gongzhongqiang 于2024年4月23日周二 16:54写道：
> Hi,
>
&
Hi Phil,
You can check my github link
<https://github.com/bgeng777/pyflink-learning/tree/main/pyflink-using-docker>
for a detailed tutorial and example codes :).
Best,
Biao Geng
Phil Stavridis 于2024年4月12日周五 19:10写道：
> Hi Biao,
>
> Thanks for looking into it and providing a d
he.org/flink/flink-docs-release-1.18/docs/dev/python/dependency_management/>
for
more details.
Best,
Biao Geng
Phil Stavridis 于2024年4月10日周三 22:04写道：
> Hi Biao,
>
> 1. I have a Flink client container like this:
> # Flink client
> flink_client:
> container_name: flink_client
&g
t;
for
more details. In short, your yaml shows that you are using the session
mode, which needs connector jar to generate job graph in the client side.
Best,
Biao Geng
Phil Stavridis 于2024年4月10日周三 18:14写道：
> Hi Biao,
>
> For submitting the job, I run t_env.execute_sql.
> Shouldn’t
hich is exactly
what you have prepared. For pyflink's table API, you can have a look at
this document:
https://nightlies.apache.org/flink/flink-docs-master/api/python/examples/table/word_count.html
Best,
Biao Geng
Phil Stavridis 于2024年4月10日周三 03:10写道：
> Hello,
>
> I have set up Fli
clean codes, in flink's abstraction, open() method
is designed for one time setup work. So it is usually better to mark these
fields as transient and initialize these fields in open() methods
(especially when we need to do some extra work like creating db connection).
Hope it helps!
Best,
Biao G
ils.
Best,
Biao Geng
Marco Villalobos 于2024年4月8日周一 09:22写道：
> Hi Lei,
>
> Have you tried enabling these Flink configuration properties?
>
> Configuration
> <https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#rocksdb-native-metrics&g
-docs-master/api/python/examples/table/basic_operations.html
https://github.com/apache/flink/blob/master/flink-python/pyflink/table/tests/test_join.py
Hope it helps!
Best,
Biao Geng
Fokou Toukam, Thierry 于2024年4月2日周二
15:41写道：
> Hi,
>
> i have 2 streams as sean in this example (*6&
ck.
Could you please show your codes with the ' !list.isEmpty() ' check?
Best,
Biao Geng
Fidea Lidea 于2024年4月2日周二 02:06写道：
> Hi Team,
>
> I have written a Flink Job which reads data in a List & then converts it
> to stream.
> *Example*:
>
> public static vo
duling strategy, the final distribution of apps after some
time is different.
Best,
Biao Geng
From: Lu Niu
Date: Thursday, September 7, 2023 at 12:17 AM
To: Geng Biao
Cc: Chen Zhanghao , Weihua Hu
, Kenan Kılıçtepe , user
Subject: Re: Uneven TM Distribution of Flink on YARN
Hi, Thanks for all you
jeprof.
[1]
https://erikwramner.files.wordpress.com/2017/10/native-memory-leaks-in-java.pdf
[2] https://www.evanjones.ca/java-native-leak-bug.html
Best,
Biao Geng
Kenan Kılıçtepe 于2023年9月6日周三 20:32写道：
> Hi,
>
> I have Flink 1.16.2 on a single server with 64GB Ram.
>
tes-deployment-how-to-provide-s3-credentials-from-hashicorp-vault>
for
more detailed instructions.
Besides, it should be possible to override Configuration object in your job
code. Are you using Application mode to run the job?
Best regards,
Biao Geng
Anuj Jain 于2023年5月8日周一 13:55写道：
> Hi Commu
xample, you actually implement an async user-defiend
table function(see here
<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#table-functions>
for
more details about UDTF usage), not a scalar UDF. The usage could be
different.
Best,
Biao Geng
Giannis Pol
can find some insights and then go to check
relevant codes.
Best,
Biao Geng
neha goyal 于2023年4月21日周五 18:42写道：
> Hello,
> I want to understand the reason behind the different behavior of Flink
> pipelines. When I am running a Kafka in and delta out pipeline with fewer
> resources tha
ngle event, it should
fulfill both of the 2 conditions). You can check if that is what you really
want.
Best,
Biao Geng
Ana Gómez González 于2023年4月19日周三 05:09写道：
> Here's a link for the previous code in a gist so you don't struggle with
> the format. Sorry.
>
>
the cost. When FLIP-271
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling> of
Autoscaling is completed, maybe Flink on K8s will have some killer
advantage.
Best,
Biao Geng
Ruibin Xing 于2022年12月19日周一 14:54写道：
> Hi all,
>
> We are currently setting up a
Congratulations!
Thanks a lot for the awesome work!
Best regards,
Biao Geng
Őrhidi Mátyás 于2022年12月15日周四 01:30写道：
> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.3.0.
>
> Release highlights:
>
>- Upgrade to Fabri
Hi Leon,
Can you share your full command for submission?
Best,
Biao Geng
Leon Xu 于2022年12月1日周四 06:27写道：
> Hi Flink Users,
>
> We ran into java.lang.ClassCastException after moving the flink job from
> session mode to application mode.
>
>
> *java.lang.ClassCastExceptio
creation which if you want to use. Also, if you want to
uninstall the operator, you should clear those resources by hand. It is not
very hard but does require some hand work.
Best,
Biao Geng
获取 Outlook for iOS<https://aka.ms/o0ukef>
发件人: Mark Lee
发送时间:
+1 to Yang's suggestion.
Yang Wang 于2022年10月26日周三 20:00写道：
> Maybe we could change the values of *taskmanager.numberOfTaskSlots* and
> *parallelism.default
> *in flink-conf.yaml of Kubernetes operator to 1, which are aligned with
> the default values in Flink codebase.
>
>
> Best,
> Yang
>
> Gy
You are right.
It contains the python package `pyflink` and some dependencies like py4j
and cloudpickle but does not contain all relevant dependencies(e.g.
`google.protobuf` as the error log shows, which I also reproduce in my own
machine).
Best,
Biao Geng
Levan Huyen 于2022年10月20日周四 19:53写道
fer to
this doc
<https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/faq/#preparing-python-virtual-environment>
for details.
Best,
Biao Geng
Levan Huyen 于2022年10月20日周四 14:11写道：
> Hi Biao,
>
> Thanks for your help. That solved my issue. It turned out th
tandalone cluster, it can work as well.
Best,
Biao Geng
Levan Huyen 于2022年10月19日周三 17:07写道：
> Hi,
>
> I'm new to PyFlink, and I couldn't run a basic example that shipped with
> Flink.
> This is the command I tried:
>
> ./bin/flink run -py examples/python/datastr
ore a minimal example.
Best,
Biao Geng
Lijie Wang 于2022年8月2日周二 09:49写道：
> Hi,
> I think the difference between ApplicationMode and PerJob is just where
> the main method is executed (ApplicationMode executes on JM, PerJob
> executes on client side). So I think your original job
ers of jars in classpath while it is also
important to package the jars more carefully to avoid the conflicts.
Best,
Biao Geng
Leon Xu 于2022年6月5日周日 03:21写道：
> Hi Flink Community,
>
> We are building on top of *org.apache.flink.yarn.YarnClusterDescriptor *to
> submit a flink applicat
lication to be redirected to Flink web ui.
Best,
Biao Geng
Shengkai Fang 于2022年5月20日周五 10:59写道：
> Hi.
>
> I am not familiar with the YARN application mode. Because the job manager
> is started when submit the jobs. So how can users know the address of the
> JM? Do we need to loo
*-jar-with-dependencies.jar
Note, you may need to substitute ververica-connector-kafka with open
sourced flink-sql-connector-kafka jar.
For your questions, to my best knowledge, '-j' and '-l' options are the
only options for now. Maybe others in the community can provide more
info
hink it is worthwhile to consider
dependency management more carefully before we make decisions.
Best,
Biao Geng
Konstantin Knauf 于2022年1月13日周四 16:32写道：
> Hi everyone,
>
> I would like to discuss and understand if the benefits of having Per-Job
> Mode in Apache Flink
44 matches
Mail list logo
