k job.
>
> I hope that helps :)
> Gyula
>
> On Thu, Jul 7, 2022 at 7:42 AM bat man wrote:
>
>> Hi,
>>
>> I want to understand the difference between session mode and the new
>> deployment mode - Flink Session Job which I believe is newly introduced as
>
Hi,
I want to understand the difference between session mode and the new
deployment mode - Flink Session Job which I believe is newly introduced as
part of the Flink Operator(1.15) release.
What's the benefit of using this mode as opposed to session mode as both
run sessions to which flink jobs ca
Has anyone experienced this or has any clue?
On Tue, Jun 14, 2022 at 6:21 PM bat man wrote:
> Hi,
>
> We are using flink 12.1 on AWS EMR. The job reads the event stream and
> enrich stream from another topic.
> We extend AssignerWithPeriodicWatermarks to assign waterma
Hi,
We are using flink 12.1 on AWS EMR. The job reads the event stream and
enrich stream from another topic.
We extend AssignerWithPeriodicWatermarks to assign watermarks and extract
timestamp from the event and handle idle source partitions.
AutoWatermarkInterval set to 5000L.
The timestamp extr
On Mon, Nov 22, 2021 at 7:26 PM Matthias Pohl
wrote:
> Hi bat man,
> this feature seems to be tied to a certain AWS SDK version [1] which you
> already considered. But I checked the version used in Flink 1.13.1 for the
> s3 filesystem. It seems like the version that's used (
Hi,
I am using flink 1.13.1 to use checkpointing(RocksDB) on s3 with native
kubernetes.
Passing in this parameter to job -
*-Dfs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider*
I am getting this error in job-manager logs -
*Caused by: com.amazonaws.AmazonClie
Hi Dennis,
Were you able to use checkpointing on s3 with native kubernetes. I am using
flink 1.13.1 and did tried your solution of passing the
webidentitytokencredentialsprovider.
*-Dfs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider*
I am getting this error in
于2021年9月14日周二 下午4:48写道：
>
>> Hi
>>
>> Maybe you could try the `kubectl describe pod -n ${namespace}
>> ${podname}` to see what happened atm.
>>
>> Best,
>> Guowei
>>
>>
>> On Tue, Sep 14, 2021 at 2:58 PM bat man wrote:
>>
>>
;
> On Mon, Sep 13, 2021 at 10:59 PM bat man wrote:
>
>> Hi,
>>
>> I am running a POC to evaluate Flink on Native Kubernetes. I tried
>> changing the default log location by using the configuration -
>> kubernetes.flink.log.dir
>> However, the job in applica
Hi,
I am running a POC to evaluate Flink on Native Kubernetes. I tried changing
the default log location by using the configuration -
kubernetes.flink.log.dir
However, the job in application mode fails after bringing up the task
manager. This is the command I use -
./bin/flink run-application --t
Hi,
I am using flink 12.1 to consume data from kafka in a streaming job. Using
the flink-connector-kafka_2.12:1.12.1. Kafka broker version is 2.2.1
In logs I see warnings like this -
2021-08-26 13:36:49,903 WARN
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher [] -
Committing o
e/KAFKA-4090
> ------
> *From:* bat man
> *Sent:* Wednesday, July 14, 2021 10:55:54 AM
> *To:* Timo Walther
> *Cc:* user
> *Subject:* Re: High DirectByteBuffer Usage
>
> Hi Timo,
>
> I am looking at these options.
> However, I had a couple of questions -
&
ble/#container-memory-exceeded
>
> It is likely that the high number of output streams could cause your
> issues.
>
> Regards,
> Timo
>
>
>
>
> On 14.07.21 08:46, bat man wrote:
> > Hi,
> > I have a job which reads different streams from 5 kafka topics. It
.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#kubernetes
> [2].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
> [3].
> https://ci.apache.org/projects/flink/
flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#fields-overwritten-by-flink
> [3].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/metric_reporters/#prometheus
>
> Best,
> Yang
>
>
> bat man 于2021年7月11日周日 下午5:27写道：
>
>&
Hi,
I am running a POC to run flink 1.13 on Native Kubernetes. Per
documentation [1] this supports custom pod templates. Are there any
references for sample pod templates which can be used as a baseline to
build upon. Could not find any references, documentation[1] has one sample
for pod-template.
erties": {
> "yarn.nodemanager.pmem-check-enabled": "false",
> "yarn.nodemanager.vmem-check-enabled": "false"
> }
> }
> ]
>
>
> …might help.
>
> — Ken
>
>
> On Jul 2, 2021, at 8:36 AM, bat man wrote:
>
&g
Hi,
I am running a streaming job (Flink 1.9) on EMR on yarn. Flink web UI or
metrics reported from prometheus shows total memory usage within specified
task manager memory - 3GB.
Metrics shows below numbers(in MB) -
Heap - 577
Non Heap - 241
DirectMemoryUsed - 852
Non-heap does rise gradually, s
Anyone who has tried this or can help on this.
Thanks.
On Thu, May 6, 2021 at 10:34 AM bat man wrote:
> Hi Users,
>
> Is there a way that Flink 1.9 the checkpointed data can be read using the
> state processor api.
> Docs [1] says - When reading operator state, users specify the
Hi Users,
Is there a way that Flink 1.9 the checkpointed data can be read using the
state processor api.
Docs [1] says - When reading operator state, users specify the operator
uid, the state name, and the type information.
What is the type for the kafka operator, which needs to be specified whil
to Table API/SQL and use SQL to query/analyse the state.
>
> Best
> Piotrek
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> śr., 14 kwi 2021 o 11:25 bat man napisał(a):
>
>> Hi All,
>>
>> Is there any
Hi All,
Is there any way I can inspect/query the checkpointed data. Scenario is
like this -
We have a high volume of data coming in the data stream pipeline for which
kafka is source, in case if fails bcoz of bad data I want to analyse the
data which caused the issue. It could be that some data s
l whether the fix can be backported.
>
> Cheers,
> Till
>
> On Thu, Apr 8, 2021 at 10:42 AM bat man wrote:
>
>> Anyone who has faced similar issues with cdc with Postgres.
>>
>> I see the restart_lsn and confirmed_flush_lsn constant since the snapshot
>>
Thanks.
On Wed, Apr 7, 2021 at 12:51 PM bat man wrote:
> Hi there,
>
> I am using flink 1.11 and cdc connector 1.1 to stream changes from a
> postgres table. I see the WAL consumption is increasing gradually even
> though the writes to tables are very less.
>
> I am usin
Hi there,
I am using flink 1.11 and cdc connector 1.1 to stream changes from a
postgres table. I see the WAL consumption is increasing gradually even
though the writes to tables are very less.
I am using AWS RDS, from [1] I understand that setting the parameter
heartbeat.interval.ms solves this W
Hi,
I am using this library -* jasync-postgresql* [1] for async calls to
postgres in asyncio operator. I am using running Flink 1.9 on EMR.
I am facing this error -
Caused by: java.lang.ClassCastException:
io.netty.channel.epoll.EpollEventLoopGroup cannot be cast to
io.netty.channel.MultithreadEv
the configuration ？
>
> For the case of configuring `env.java.opts`, it seems the JobManager also
> could not be launched with
> this configuration.
>
> Best,
> Yun
>
> ----------Original Mail --
> *Sender:*bat man
> *Send Date:*Sat Mar 6 16
Mar 2021 at 7:48 AM, Xintong Song wrote:
> Hi Hemant,
> I don't see any problem in your settings. Any exceptions suggesting why TM
> containers are not coming up?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Sat, Mar 6, 2021 at 3:53 PM bat man wrote:
>
>
Hi,
I am trying to generate a heap dump to debug a GC overhead OOM. For that I
added the below java options in flink-conf.yaml, however after adding this
the yarn is not able to launch the containers. The job logs show it goes on
requesting for containers from yarn and it gets them, again releases
t; memory usages leveraging profiling tools.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/m
Hi,
Getting the below OOM but the job failed 4-5 times and recovered from there.
j
*ava.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit
exceededat
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(S
gt; its state, and re-emits it upon recovery.
>
> Best,
> Piotrek
>
> czw., 4 mar 2021 o 15:43 bat man napisał(a):
>
>> Hi All,
>>
>> I have a job where my source is kafka. Stream1 is partition the data on
>> dynamic key, join the data with static rules(source ka
Hi All,
I have a job where my source is kafka. Stream1 is partition the data on
dynamic key, join the data with static rules(source kafka).I use
KeyedCoProcessFunction to join the Steam1 with Stream2(source kafka). All
works fine in a normal run.
For changing the watermark generation interval I s
uot;).uid("rt").setParallelism(5);
On Fri, Feb 26, 2021 at 3:38 PM Arvid Heise wrote:
> Hi,
>
> I have no idea what's going on. There is no mechanism in DataStream to
> react to deleted records.
>
> Can you reproduce it locally and debug through it?
>
>
>
&
cess function [1]. If you have done that, I'm a bit lost.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html
>
> On Wed, Feb 24, 2021 at 7:30 AM bat man wrote:
>
>> Hi,
>>
>> This is my code below -
&g
ka shouldn't affect Flink state in general.
> Probably, some operator in your pipeline is re-reading the topic
> and overwrites the state, dropping what was deleted by Kafka.
> Could you share the code?
>
> Regards,
> Roman
>
>
> On Tue, Feb 23, 2021 at 7:12 AM bat ma
Hi,
I have 2 streams one event data and the other rules. I broadcast the rules
stream and then key the data stream on event type. The connected stream is
processed thereafter.
We faced an issue where the rules data in the topic got deleted because of
Kafka retention policy.
Post this the existing
we don't differentiate between the two when it
> comes to metrics.
>
> On 2/19/2021 3:59 AM, bat man wrote:
>
> I meant the Flink jobname. I’m using the below reporter -
>
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
>
> Is there
hen you mean "job_name", are you referring to the Prometheus concept of
> jobs, of the one of Flink?
>
> Which of Flink prometheus reporters are you using?
>
> On 2/17/2021 7:37 PM, bat man wrote:
> > Hello there,
> >
> > I am using prometheus to push metr
Hello there,
I am using prometheus to push metrics to prometheus and then use grafana
for visualization. There are metrics like
- flink_taskmanager_Status_JVM_CPU_Load,
flink_taskmanager_Status_JVM_CPU_Load,
flink_taskmanager_Status_JVM_CPU_Time
etc which do not gives job_name. It is tied to an in
ant,
>
> Have you tried running a new Flink version? Can you create a heap dump
> when the process fails? This could help us digging into whether there is
> some memory leak.
>
> Cheers,
> Till
>
> On Tue, Feb 16, 2021 at 5:21 PM bat man wrote:
&g
Hi there,
I am facing *java.lang.OutOfMemoryError: GC overhead limit exceeded *when
using prometheus exporter with* Flink 1.9 *on *AWS EMR *emr-5.28.1. I have
other jobs which run fine. tihs specific job fails with the below error
stack.
Exception in thread "pool-3-thread-2" java.lang.OutOfMemory
I was able to make it work with a fresh Elastic installation. Now
taskmanager and jobmanager logs are available in elastic.
Thanks for the pointers.
-Hemant.
On Wed, Jan 13, 2021 at 6:21 PM Aljoscha Krettek
wrote:
> On 2021/01/11 01:29, bat man wrote:
> >Yes, no entries to the elast
be located in your console output and the "log/"
> directory of your Flink distribution.
>
> Best,
> Yangze Guo
>
> On Tue, Jan 12, 2021 at 4:50 PM bat man wrote:
> >
> > Hi,
> >
> > I am running a sample job as below -
> >
> >
Hi,
I am running a sample job as below -
public class WordCountExample {
static Logger logger = LoggerFactory.getLogger(WordCountExample.class);
public static void main(String[] args) throws Exception {
final ExecutionEnvironment env = ExecutionEnvironment.
getExecutionEnvironment();
logger.inf
Hi Team,
I have a requirement to push the flink app logs to Elastic Search for log
management. Can anyone guide if you are already doing this.
I have tried this -
https://cristian.io/post/flink-log4j/
I’m not getting any error for a sample job I tried.
I am using EMR to run Flink 1.9 and Elastic
influxdb.username:
> flink-metricsmetrics.reporter.influxdb.password:
> qwertymetrics.reporter.influxdb.retentionPolicy: one_hour
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#influxdb-orgapacheflinkmetricsinfluxdbinfluxdbreporter
>
> On Thu, Aug
Anyone who has made metrics integration to external systems for flink
running on AWS EMR, can you share if its a configuration issue or EMR
specific issue.
Thanks,
Hemant
On Wed, Aug 12, 2020 at 9:55 PM bat man wrote:
> An update in the yarn logs I could see the below -
>
> Classpat
he properties and metrics reported but
then org.apache.flink.runtime.metrics.ReporterSetup complains of not
finding it.
Can anyone guide what I am missing here.
Thanks,
Hemant
On Wed, Aug 12, 2020 at 9:15 PM bat man wrote:
> Hello Experts,
>
> I am running Flink - 1.9.0 on AWS EMR(emr-5.
Hello Experts,
I am running Flink - 1.9.0 on AWS EMR(emr-5.28.1). I want to push
metrics to Influxdb. I followed the documentation[1]. I added the
configuration to /usr/lib/flink/conf/flink-conf.yaml and copied the jar to
/usr/lib/flink//lib folder on master node. However, I also understand that
t
ttps://github.com/ververica/flink-training-troubleshooting/blob/master/src/main/java/com/ververica/flinktraining/solutions/troubleshoot/TroubledStreamingJobSolution2.java#L129-L187
>
> On Thu, Jul 23, 2020 at 8:48 PM bat man wrote:
>
>> Thanks Niels for a great talk. You have covered
the next stream(next kafka topic) after enrichment. Have you got any
solution for this?
-Hemant
On Thu, Jul 23, 2020 at 12:05 PM Niels Basjes wrote:
> Have a look at this presentation I gave a few weeks ago.
> https://youtu.be/bQmz7JOmE_4
>
> Niels Basjes
>
> On Wed, 22 Jul 2
Hi Team,
Can someone share their experiences handling this.
Thanks.
On Tue, Jul 21, 2020 at 11:30 AM bat man wrote:
> Hello,
>
> I have a pipeline which consumes data from a Kafka source. Since, the
> partitions are partitioned by device_id in case a group of devices is down
>
Hello,
I have a pipeline which consumes data from a Kafka source. Since, the
partitions are partitioned by device_id in case a group of devices is down
some partitions will not get normal flow of data.
I understand from documentation here[1] in flink 1.11 one can declare the
source idle -
Watermar
54 matches
Mail list logo
