/connectors/table/filesystem/
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/
Best,
Shammon FY
On Mon, Sep 25, 2023 at 12:36 PM Kirti Dhar Upadhyay K <
kirti.k.dhar.upadh...@ericsson.com> wrote:
> Thanks Shammon.
>
> Is there any way to verify t
job is
completed, the directory will be cleaned up.
Best,
Shammon FY
On Fri, Sep 22, 2023 at 3:11 PM Kirti Dhar Upadhyay K via user <
user@flink.apache.org> wrote:
> Hi Community,
>
>
>
> I am using Flink File Source with Amazon S3.
>
> Please help me on below question
Hi Yifan,
Besides reading job state, I would like to know what statebackend are you
using? Can you give the configurations about state and checkpoint for your
job? Maybe you can check these configuration items to confirm if they are
correct first.
Best,
Shammon FY
On Wed, Sep 6, 2023 at 3:17 PM
.
Best,
Shammon FY
On Sat, Sep 2, 2023 at 6:26 AM David Anderson wrote:
> This may or may not help, but you can get the execution plan from
> inside the client, by doing something like this (I printed the plan to
> stderr):
>
> ...
> System.err.println(en
atement in your application.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#terminating-a-job
Best,
Shammon FY
On Fri, Sep 1, 2023 at 6:35 AM Krzysztof Chmielewski <
krzysiek.chmielew...@gmail.com> wrote:
> Hi community,
> I would like to ask what is
Hi,
I seems that the node `tef-prod-flink-04/10.11.0.51:37505 [
tef-prod-flink-04:38835-e3ca4d ]` exits unexpected, you can check whether
there are some errors in the log of TM or K8S
Best,
Shammon FY
On Sun, Aug 20, 2023 at 5:42 PM Kenan Kılıçtepe
wrote:
> Hi,
>
> Nothing inter
eck whether the dispatcher is
started successfully.
Best,
Shammon FY
On Wed, Aug 16, 2023 at 2:55 PM elakiya udhayanan
wrote:
> Hi Shammon,
>
> Thanks for your response.
>
> If it is a network issue as you have mentioned, how does it read the
> contents of the jar file, we can
configure `web.submit.enable=false` in your cluster, you could not
upload a jar job, but you can still submit jobs via rest endpoint. You can
create your `RestClusterClient` to do that or using the existing
jdbc-driver and sql-gateway.
Best,
Shammon FY
On Tue, Aug 15, 2023 at 12:14 AM patricia lee
Hi,
It seems that the client can not access the right network to submit you
job, maybe the address option in k8s is wrong and you can check the error
message in k8s log
Best,
Shammon FY
On Fri, Aug 11, 2023 at 11:40 PM elakiya udhayanan
wrote:
>
> Hi Team,
> We are using Apache Fli
]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/
Best,
Shammon FY
On Thu, Aug 10, 2023 at 2:43 PM liu ron wrote:
> Hi, lvan
>
> You can refer to the five-part that restore the job in [1].
>
> [1]
> https://nightlies.apache.org/flink/flink-k
;
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY
> *Sent:* 03 August 2023 10:54 AM
> *To:* Kamal Mittal
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> I think it depends on your pr
. However, this approach will increase data latency. The data is only
visible after the checkpoint is completed and the data is committed, rather
than immediately visible after the sink writes the data.
Best,
Shammon FY
On Thu, Aug 3, 2023 at 12:23 PM Kirti Dhar Upadhyay K via user <
u
. If
they use the same port, it may also cause conflicts.
Best,
Shammon FY
On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal
wrote:
> Hello Shammon,
>
>
>
> Please have a look for below and share views.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Kamal Mitta
Hi arjun,
As @Mate mentioned, the discussion of FLIP-314 has been completed and a
vote will be initiated soon. We would like to introduce the interfaces for
lineage in the next release of Flink after 1.18
Best,
Shammon FY
On Tue, Aug 1, 2023 at 11:07 PM Mate Czagany wrote:
>
of
tasks per TM also limits the number of slots.
Best,
Shammon FY
On Tue, Aug 1, 2023 at 4:22 PM liu ron wrote:
> Hi, Kamal
>
> > How many task managers a job manager can handle? Is there any upper
> limit also?
>
> There is no clear limit to how many TMs a JM can
urceReader which will read data from netty source
according to the splits.
Best,
Shammon FY
On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user
wrote:
> Hello,
>
>
>
> I created a custom server socket source in which opened server socket in
> split enumerator, source pa
I cannot find any information about netty source in flink website and it is
not in the connector list[1], so I'm think that it is not supported by
flink community
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
Best,
Shammon FY
On Thu, Jul 27,
Hi Kamal,
There's socket text stream in `DataStream` and you can refer to [1] for
more details.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
Best,
Shammon FY
On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user
wrote:
&g
Hi nick,
Is there any error log? That may help to analyze the root cause.
On Sun, Jul 23, 2023 at 9:53 PM nick toker wrote:
> hello
>
>
> we replaced deprecated kafka producer with kafka sink
> and from time to time when we submit a job he stack for 5 min in
> inisazaing ( on sink operators)
>
rt and this limits this
> source task to execute only at one task manager which we want to avoid and
> want to scale it across task managers.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY
> *Sent:* 21 July 2023 07:41 AM
> *To:* Kamal Mittal
th min
utilization.
Back to your example, if all slots in TM1 and TM2 match the task
requirements, there will be one free slot in TM1 and TM2 after your job is
scheduled if `cluster.evenly-spread-out-slots` is set to true.
Best,
Shammon FY
On Thu, Jul 20, 2023 at 3:47 PM Kamal Mittal via user
-master/docs/dev/table/sql-gateway/hiveserver2/
[3]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/
Best,
Shammon FY
On Thu, Jul 20, 2023 at 12:48 AM Rajat Ahuja
wrote:
> Hi team,
>
> I have set up a session cluster on k8s via sql gateway. I am looking
ith regular agg operators, and observe whether the checkpoint and
savepoint meet expectations.
Best,
Shammon FY
On Tue, Jul 18, 2023 at 8:25 PM Neha . wrote:
> Hi Shammon,
>
> These configs exist in Flink WebUI. We have set
> exeEnv.setStreamTimeCharacteristic(TimeCharacteristic.IngestionT
-master/docs/dev/table/sqlclient/#start-a-sql-job-from-a-savepoint
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#terminating-a-job
Best,
Shammon FY
On Tue, Jul 18, 2023 at 9:56 AM Xiaolong Wang
wrote:
> Hi, Shammon,
>
> I know that the job manager can aut
peedup the
lookup request which we always do in loopup join for sql jobs.
Best,
Shammon FY
On Mon, Jul 17, 2023 at 10:09 PM Pritam Agarwala <
pritamagarwala...@gmail.com> wrote:
> Hi Team,
>
>
> Any input on this will be really helpful.
>
>
> Thanks!
>
> On Tu
Hi Neha,
I think you can first check whether the options `state.backend` and
`state.backend.incremental` you mentioned above exist in
`JobManager`->`Configuration` in Flink webui. If they do not exist, you may
be using the wrong conf file.
Best,
Shammon FY
On Mon, Jul 17, 2023 at 5:04 PM N
Hi Neha,
I noticed that the `Checkpointed Data Size` is always equals to `Full
Checkpoint Data Size`, I think the job is using full checkpoint instead of
incremental checkpoint, you can check it
Best,
Shammon FY
On Mon, Jul 17, 2023 at 10:25 AM Neha . wrote:
> Hello Shammon,
>
>
occurs.
Best,
Shammon FY
On Thu, Jul 13, 2023 at 10:22 AM Xiaolong Wang
wrote:
> Hi,
>
> I'm currently working on providing a SQL gateway to submit both streaming
> and batch queries.
>
> My question is, if a streaming SQL is submitted and then the jobmanager
> crashes
Hi Eugenio,
I cannot catch it clearly, could you describe it in more detail?
Best,
Shammon FY
On Sat, Jul 15, 2023 at 5:14 PM Eugenio Marotti <
ing.eugenio.maro...@gmail.com> wrote:
> Hi everyone,
>
> is there a way to set Flink processing time in the past?
>
> Thanks
> Eugenio
>
-master/docs/ops/state/state_backends/#incremental-checkpoints
Best,
Shammon FY
On Sun, Jul 16, 2023 at 2:30 PM Neha . wrote:
> Hello Shammon FY,
>
> It is a production issue for me. Can you please take a look if anything
> can be done?
>
> -- Forwarded message
Hi Anastasios,
I think you may need to implement a customized trigger to emit record when
a session window is created. You can refer to [1] for more detailed
information.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#triggers
Best,
Shammon FY
]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-280%3A+Introduce+EXPLAIN+PLAN_ADVICE+to+provide+SQL+advice
Best,
Shammon FY
On Tue, Jul 11, 2023 at 3:20 PM amenreet sodhi wrote:
> Hi Team,
>
> I wanted to confirm, the local state which TM stores in the directory
> eithe
Hi amenreet,
According to the error message, I think you can log in the jm pod after it
starts, and check access permissions for the directory
`file:///opt/flink/pm/ha`
Best,
Shammon FY
On Fri, Jul 7, 2023 at 6:04 PM amenreet sodhi wrote:
> Hi Shammon
>
> I am using an external
Hi amenreet,
Maybe you can try to use hdfs or s3 for `high-availability.storageDir`, I
found your current job is using a local file which is started with
`file:///`.
Best,
Shammon FY
On Fri, Jul 7, 2023 at 4:20 PM amenreet sodhi wrote:
> Hi All,
> I am deploying Flink cluster on Kube
Hi Xiaolong,
I think you may need to check the error log in the flink cluster to find
out the root cause.
Best,
Shammon FY
On Tue, Jul 4, 2023 at 3:38 PM Xiaolong Wang
wrote:
> The flink web ui is fine until I run the Hive query. After that the flink
> deployment is down and the web UI
Hi Xiaolong,
>From the exception it seems that the flink session cluster is not
running properly. Can you visit the flink web ui and everything is ok?
Best,
Shammon FY
On Mon, Jul 3, 2023 at 2:43 PM Xiaolong Wang
wrote:
> Hi,
> I've tested the Flink SQL-gateway to run some simpl
Hi Kamal,
You can check whether flink job has readed data from source in flink web
ui, it will show total record count and size for each operator.
Best,
Shammon FY
On Sat, Jul 1, 2023 at 4:53 PM Kamal Mittal via user
wrote:
> Hello Community,
>
>
>
> I have a requirement to r
/#switching-the-memory-allocator
Best,
Shammon FY
On Sat, Jul 1, 2023 at 2:58 PM neha goyal wrote:
> Hello,
>
> I am trying to debug the unbounded memory consumption by the Flink
> process. The heap size of the process remains the same. The size of the RSS
> of the process keeps o
Hi Mahmoud,
For the third quest, currently flink uses Fine-Grained Resource Management
to choose a TM for tasks, you can refer to the doc [1] for more information.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/finegrained_resource/
Best,
Shammon FY
On Thu, Jun 29
Hi Brendan,
I think you may need to confirm which stage the job is blocked, the client
is submitting job or resourcemanage is scheduling job or tasks are
launching in TM? May be you need provide more information to help us to
figure the issue
Best,
Shammon FY
On Tuesday, June 27, 2023, Weihua
Hi elakiya,
I think you may need to spread the columns in key and value format, then
you can use the specific column as a primary key in the ddl.
Best,
Shammon FY
On Fri, Jun 23, 2023 at 6:36 PM elakiya udhayanan
wrote:
> Hi team,
>
> I have a Kafka topic named employee which uses
/finegrained_resource/
Best,
Shammon FY
On Wed, Jun 21, 2023 at 3:18 AM Pritam Agarwala
wrote:
> Hi Team,
>
> I have a confusion about how TM's core will be used for task slots.
>
> I have TMs with 3 cores and 4 task slots . Each task slot will be using
> one core to perform a
r the
checkpoint.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/large_state_tuning/#incremental-checkpoints
Best,
Shammon FY
On Tue, Jun 20, 2023 at 4:50 PM Alex Nitavsky
wrote:
> Hello Prabhu,
>
> On your place I would check:
>
> 1. That there is no
Hi Anastasios,
What you want sounds like a session window [1], maybe you can refer to the
doc for more details.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#session-windows
Best,
Shammon FY
On Thu, Jun 15, 2023 at 10:03 PM Anastasios Makris
,
Shammon FY
On Tue, Jun 13, 2023 at 11:13 AM Kamal Mittal via user <
user@flink.apache.org> wrote:
> Hello,
>
>
>
> Using Flink record stream format file source API as below for parquet
> records reading.
>
>
>
> FileSource.FileSourceBuilder source = FileSource.
&g
Hi Yogesh,
I think you need to build the dynamic SQL statement in your service and
then submit the SQL to flink cluster.
Best,
Shammon FY
On Mon, Jun 12, 2023 at 9:15 PM Yogesh Rao wrote:
> Hi,
>
> Is there a way we can build a dynamic SQL in Flink from contents of Map ?
>
&
ire(CompletableFuture.java:1072)
It seems that the client tried to submit a job to the flink cluster through
the rest api failed, maybe you need to provide more information such as
config of k8s for the job and community can help better analyze problems.
Best,
Shammon FY
On Wed, Jun 7, 2023 at
try to fix this issue.
[1] https://github.com/apache/flink/pull/22718
[2] https://issues.apache.org/jira/browse/FLINK-32265
Best,
Shammon FY
On Tue, Jun 6, 2023 at 8:58 AM Li Shao wrote:
> Hi Shammon,
>
> Thank you for your reply. My flink job is using batch mode. For streaming
>
? Thanks
Best,
Shammon FY
On Tue, Jun 6, 2023 at 4:27 AM Li Shao wrote:
> Hi,
>
> Recently I noticed my job manager JVM metaspace is keeping increasing for
> running batch flink jobs. I found similar stackoverflow post:
> https://stackoverflow.com/questions/73184042/apache-flink-job-ma
Hi Ryan,
What I usually encounter is writing Protobuf format data to systems such as
Kafka, and I have never encountered writing to a file yet.
Best,
Shammon FY
On Mon, Jun 5, 2023 at 10:50 PM Martijn Visser
wrote:
> Hey Ryan,
>
> I've never encountered a use case for wr
Hi Kamal,
The network buffer will be full for specific `FileSource` when the job has
back pressure which will block the source subtask. You can refer to network
buffer [1] for more information.
[1]
https://flink.apache.org/2019/06/05/a-deep-dive-into-flinks-network-stack/
Best,
Shammon FY
On
connection url
Best,
Shammon FY
On Thu, May 25, 2023 at 8:32 AM Hemi Grs wrote:
> hey everybody,
>
> I have a problem with my apache flink, I am synchronizing from MySQL to
> Elasticsearch but it seems that I can't run more than 19 tasks. it gave me
> this
wrote:
>
>> Added Flink community DL as well.
>>
>> -- Forwarded message -
>> From: Kamal Mittal
>> Date: Tue, May 23, 2023 at 7:57 AM
>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
>> To: Shammon FY
>>
>
Hi Kamal,
If I understand correctly, do you want the source to do some custom
actions, such as current limiting, when there is backpressure in the job?
Best,
Shammon FY
On Mon, May 22, 2023 at 2:12 PM Kamal Mittal wrote:
> Hello Community,
>
> Can you please share views about the qu
Hi Sharif,
I would like to know what do you want to do with the exception after
catching it? There are different ways for different requirements, for
example, Flink has already reported these exceptions.
Best,
Shammon FY
On Mon, May 22, 2023 at 4:45 PM Sharif Khan via user
wrote:
>
s a
`List partitions` field in `PartitionCommitInfo` which will cause
the exception you mentioned in the thread. I have created an issue [1] for
this.
[1] https://issues.apache.org/jira/browse/FLINK-32129
Best,
Shammon FY
On Thu, May 18, 2023 at 9:20 PM Aniket Sule
wrote:
> Hi,
>
>
POINTWISE, JobManager will create independent relations between subtasks
and put the related subtasks to one region.
Best,
Shammon FY
On Tue, May 16, 2023 at 6:42 PM Prabhu Joseph
wrote:
> Yes i expected the same. But all the tasks goes into one region and
> RestartPipelinedRegionFailoverSt
n IDE directly after the planner
dependency is added.
Best,
Shammon FY
On Mon, May 15, 2023 at 10:18 PM Lucifer_jl wrote:
> hi:
> I think there may be the following reasons：
> 1、the scope of dependency is 'provided' in your "pom.xml" 。When you
> package, this
Hi Ajinkya,
The command 'jcmd GC.run' can be used to trigger fullgc for JVM
process. However, it should be noted that this may have a performance
impact on the ongoing computation.
Best,
Shammon FY
On Mon, May 15, 2023 at 10:51 AM Ajinkya Pathrudkar <
ajinkya.pathrudka...@gma
Hi Ajinkya,
The memory of metaspace may need to be released through FullGC, you can try
to trigger fullgc manually in JobManager and TaskManager, and check whether
the metaspace is released.
Best,
Shammon FY
On Sat, May 13, 2023 at 4:01 PM Jiadong lu wrote:
> Hi, Ajinkya
>
> M
Hi Luke,
Maybe you can get 'JobClient' after submit the job and check the job status
with 'JobClient.getJobStatus()'
Best,
Shammon FY
On Fri, May 12, 2023 at 2:58 AM Luke Xiong wrote:
> Hi,
>
> My flink job needs to do something when the pipeline execution has
Hi Yangxueyong,
Are you sure this is your Flink SQL job? This SQL statement looks very
strange, the table 'test_flink_res2' is both source and sink, and the join
key is null.
Best,
Shammon FY
On Wed, May 10, 2023 at 12:54 PM yangxueyong
wrote:
> flink1.16.1
>
> mysql8.0.33
Hi Madan,
Could you give the old and new versions of flink and provide the job plan?
I think it will help community to find the root cause
Best,
Shammon FY
On Wed, May 10, 2023 at 2:04 AM Madan D via user
wrote:
> Hello Team,
>
> We have been using Flink Kafka consumer and recentl
7;SplitEnumerator' in JobManager will get splits from kafka and send them to
source subtask to read data.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
Best,
Shammon FY
On Tue, May 9, 2023 at 2:37 AM Wei Hou via user
wrote:
> Hi Team,
>
&g
is used in your sql?
The error message "No match found for function signature" usually indicates
that the function does not exist or the parameters do not match.
Best,
Shammon FY
On Sun, May 7, 2023 at 2:55 PM Giannis Polyzos
wrote:
> I can't really find any examples / docs f
Hi yxy,
As Hang mentioned, I think lookup join matches your requirements too. You
can refer to doc [1] to get more detailed information.
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
Best,
Shammon FY
On Sat, May 6, 2023 at 5:54 PM
can also add this package in your cluster
environment.
Best,
Shammon FY
On Fri, May 5, 2023 at 10:21 PM DEROCCO, CHRISTOPHER wrote:
> How can I add the package to the flink job or check if it is there?
>
>
>
> *From:* Shammon FY
> *Sent:* Thursday, May 4, 2023 9:59
on in this FLIP [1].
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
Best,
Shammon FY
On Fri, May 5, 2023 at 11:48 PM John Gerassimou
wrote:
> Sorry for the mix-up. I read your message wrong. Please ignore my last
> reply.
>
> On Fri, May 5, 2023 at 11
如果需要取消订阅 user@flink.apache.org 和 d...@flink.apache.org 邮件组，请发送任意内容的邮件到
user-unsubscr...@flink.apache.org 和 dev-unsubscr...@flink.apache.org ,参考[1]
[1] https://flink.apache.org/zh/community/
On Fri, May 5, 2023 at 10:43 AM 谢浩天 wrote:
> 您好：
>退订！
>
>
> 谢浩天
>
>
ssing this package.
Best,
Shammon FY
On Fri, May 5, 2023 at 4:40 AM DEROCCO, CHRISTOPHER wrote:
>
>
> I receive the error: *Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider not found.*
>
> I’m using flink 1.16 running i
Hi Luke,
This issue has been fixed in FLINK-25645 [1]. You can try 1.16 and later
versions of flink.
[1] https://issues.apache.org/jira/browse/FLINK-25645
Best,
Shammon FY
On Tue, May 2, 2023 at 12:18 PM Luke Xiong wrote:
> Many thanks if anybody could help. I ran into this in version 1
Then the streaming jobs can produce data
to different partitions.
Best,
Shammon FY
On Wed, Apr 26, 2023 at 7:22 PM Jiadong Lu wrote:
> Hi Shanmmon,
>
> Thank you for your quick response.
>
> To give you some context, I am working on a project that involves
> joining two stream
comparisons, such as this document [2], and the results are for reference
only.
[1] https://github.com/ververica/flink-sql-benchmark
[2]
https://files.alicdn.com/tpsservice/c399186c83bb17dc5dd72cbd0c6ccf71.pdf?spm=a2csy.flink.0.0.49496ea8lXSLb8&file=c399186c83bb17dc5dd72cbd0c6ccf71.pdf
Best,
Sha
Hi Jiadong
Using the process time window in Batch jobs may be a little strange for
me. I prefer to partition the data according to the day level, and then the
Batch job reads data from different partitions instead of using Window.
Best,
Shammon FY
On Wed, Apr 26, 2023 at 12:03 PM Jiadong Lu
Hi Jiadong,
I think it depends on the specific role of the window here for you. If this
window has no specific business meaning and is only used for performance
optimization, maybe you can consider to use join directly
Best,
Shammon FY
On Tue, Apr 25, 2023 at 5:42 PM Jiadong Lu wrote:
> He
hread/nbf3skopy3trtj37jcovmt6ktcgst4w8
Best,
Shammon FY
On Mon, Apr 24, 2023 at 3:30 PM Илья Соин wrote:
> Hi Shammon FY,
>
> I haven’t tried it because AFIK it’s only available in the DataStream API,
> while our job is in SQL. I’m thinking to write a custom
> HybridDynamicTableSource which will us
Hi Илья
I think HybridSource may be a good way. Have you tried it before? Or have
you encountered any problems?
Best,
Shammon FY
On Fri, Apr 21, 2023 at 5:59 PM Илья Соин wrote:
> Hi Flink community,
>
> We have a quite complex sql job, it unions 5 topics, deduplicates by key
> a
Hi Vinay
If you need to do that, maybe you should get the job client or job
execution result and check the status of job in your application
Best,
Shammon FY
On Sun, Apr 23, 2023 at 2:03 PM Weihua Hu wrote:
> Hi,
>
> The Flink client always exits when the application submission is
Hi Naga
Could you provide detailed error information? I think it may be useful for
positioning the issue
Best,
Shammon FY
On Fri, Apr 21, 2023 at 12:56 AM naga sudhakar
wrote:
> Hi Team,
> Greetings of the day..
> we are on flink 1.16.1 version and using flinkkafkaconsumer today.
Hi neha
Flink can delete runtime data for a job when it goes to termination. But
for external files such as udf jar files as you mentioned, I think you need
to manage them yourself. The files may be shared between jobs, and can not
be deleted when one flink job exists.
Best,
Shammon FY
On Wed
sword",
"secret",
"fs.azure.account.key",
"apikey",
"auth-params",
"service-key",
"token",
"basic-auth",
"ja
Hi Salva
I think you can check whether `foo.bar.Job` is in the classpath of your
job. “java.lang.NoClassDefFoundError: Could not initialize class
foo.bar.Job” error usually occurs when this class is not in the classpath
Best,
Shammon FY
On Mon, Apr 17, 2023 at 5:48 PM Salva Alcántara
wrote
Hi Si-li
Could you give some more detailed exceptions? Or you can check the metrics
of your job such as memory usage.
Best,
Shammon FY
On Fri, Apr 14, 2023 at 5:14 PM Si-li Liu wrote:
> My job read data from mysql and write to doris. It will crash after 20
> mins ~ 1 hour after
Hi
It seems your job did not connect the kafka correctly or the network
between your cluster and kafka is unreachable. I think you can get detailed
information from log or check the network
Best,
Shammon FY
On Sat, Apr 15, 2023 at 3:37 AM Lampros Makrodimitris
wrote:
> Hi community,
>
an issue https://issues.apache.org/jira/browse/FLINK-31801
Best,
Shammon FY
On Thu, Apr 13, 2023 at 6:40 PM Ruibin Xing wrote:
> Hi community,
>
> When following the instructions for importing es connector into my
> project on
> https://nightlies.apache.org/flink/flink-docs-re
.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/time/#lateness
Best,
Shammon FY
On Thu, Apr 13, 2023 at 8:27 PM Kirti Dhar Upadhyay K via user <
user@flink.apache.org> wrote:
> Hi,
>
>
>
> I am using Data stream file source connector in one of my use
Hi
The job in ns2 has the permission to stop the job in ns1? How about
managing the relationship in your `Job Submission Service` if it exists.
The service can check and stop the job in ns1 before it submitting the job
to ns2, what do you think?
Best,
Shammon FY
On Thu, Apr 13, 2023 at 10
Hi
Currently, Calcite supports standard SQL, so I think the main SQL syntax
will remain unchanged or backward compatible even when Flink upgrades its
calcite version. You can refer to it.
Best,
Shammon FY
On Tue, Apr 11, 2023 at 12:00 PM ravi_suryavanshi.yahoo.com via user <
u
/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/PojoTypeInfo.java
Best,
Shammon FY
On Wed, Apr 12, 2023 at 1:15 AM Alexis Sarda-Espinosa <
sarda.espin...@gmail.com> wrote:
> Hello,
>
> according to the documentation, a POJO must have a no-arg construct
Hi naga
Could you provide a specific description of your scene? It sounds like your
requirement requires a uniqueness check to ensure that there are no
multiple identical jobs running simultaneously, right?
Best,
Shammon FY
On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar
wrote:
> Thanks
option: state.backend
The snapshot of state can be stored in JobManager. When the state is large,
flink supports storing the snapshot a distributed storage with option:
state.checkpoints.dir:
Best,
Shammon FY
On Mon, Apr 10, 2023 at 12:31 AM Feng Jin wrote:
> Hi Dongwoo
>
>
> This
Hi neha
I think you can compare the detailed information about memory usage or
error messages. You can also dump heap memory to analyse the objects with
some tools for the two versions of flink
Best,
Shammon FY
On Thu, Apr 6, 2023 at 9:58 PM neha goyal wrote:
> Hello,
> I want to unde
Hi hjw
To rescale data for dim join, I think you can use `partition by` in sql
before `dim join` which will redistribute data by specific column. In
addition, you can add cache for `dim table` to improve performance too.
Best,
Shammon FY
On Tue, Apr 4, 2023 at 10:28 AM Hang Ruan wrote:
>
by option
`taskmanager.memory.jvm-metaspace.size`
Best,
Shammon FY
On Wed, Mar 29, 2023 at 3:04 AM tom yang wrote:
> Hi,
>
> I am running a standalone cluster setup and submit flinksql job with
> python udf following the examples here
>
>
> <https://github.com/ververica
Thanks Lasse, I think you can create an issue and update the document if
there is any wrong
Best,
Shammon FY
On Wed, Mar 29, 2023 at 3:48 AM Lasse Nedergaard <
lassenedergaardfl...@gmail.com> wrote:
> Hi.
> I have figured it out. The documentation are wrong in both places.
>
>
Hi Lasse
Does your job table/sql or datastream? Here's the doc [1] for customized
source in table and there is an example of socket source.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/
Best,
Shammon FY
On Mon, Mar 27, 2023 at 8:02 PM Lasse Neder
between the new and old versions.
Best,
Shammon FY
On Mon, Mar 27, 2023 at 10:38 PM Kiran Kumar Kathe <
kirankumarkathe...@gmail.com> wrote:
> Even though when I run with the latest versions I am not able to see the
> jobs running in the flink in resource manager Web UI , but when I use
about network buffer in doc [1]
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/
Best,
Shammon FY
On Wed, Mar 22, 2023 at 1:20 PM Le Xu wrote:
> Hello!
>
> I would like to run a legacy flink project on top of old-version Flink
Hi Ajinkya
It seems that the JobMaster is not started successfully. If possible, you
can provide more information to help analyze the problem.
Best,
Shammon FY
On Wed, Mar 22, 2023 at 10:57 AM Ajinkya Pathrudkar <
ajinkya.pathrudka...@gmail.com> wrote:
>
> I am writing to inf
/master/docs/content/docs/ops/batch/batch_shuffle.md#sort-shuffle
Best,
Shammon FY
On Tue, Mar 21, 2023 at 12:22 AM Ajinkya Pathrudkar <
ajinkya.pathrudka...@gmail.com> wrote:
>
> Ajinkya Pathrudkar
> 10:53 AM (1 hour ago)
> to user-info
> I hope this email finds you well. I am
1 - 100 of 127 matches
Mail list logo
