Hello,
Question 1
I don’t see any reference material showing how to write compressed (gzip) files
with StreamingFileSink. Can someone point me in the right direction?
Question 2
We currently have a use case for a “StreamingFileProcessFunction”. Basically we
need an output for the StreamingFileS
I'm working on getting a flink job into production. As part of the production
requirement, I need telemetry/metrics insight into my flink job. I have
followed instructions in
https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html
- Added the flink graphite jar to tas
using flink 1.5.1
J.
From: Chesnay Schepler
Sent: Friday, August 10, 2018 2:08 AM
To: John O ; user@flink.apache.org
Subject: Re: flink telemetry/metrics
What is wrong with the metrics that are shown in graphite?
Can you provide us with the metrics section of your flink-conf.yaml?
Are the
I am noticing that watermark does not progress as expected when running locally
in IDE. It just stays at Long.MIN
I am using EventTime processing and have tried both these time extractors.
* assignAscendingTimestamps ...
*
assignTimestampsAndWatermarks(BoundedOutOfOrdernessTime
watermark.
Jo
From: Fabian Hueske
Sent: Wednesday, August 15, 2018 2:07 AM
To: vino yang
Cc: John O ; user
Subject: Re: watermark does not progress
Hi John,
Watermarks cannot make progress if you have stream partitions that do not carry
any data.
What kind of source are you using?
Best,
Fab
ill have to choose which watermark to use from the multiple
sources(the lowest value) thus never progressing the watermark.
Jo
From: Hequn Cheng
Sent: Wednesday, August 15, 2018 6:38 AM
To: John O
Cc: Fabian Hueske ; vino yang ; user
Subject: Re: watermark does not progress
Hi John,
I gu
I am consuming data from a kafka topic that has multiple partitions. I would
like to keyby(record.partition). What would be the best way to get access to
the partition information?
Jo
Found it. Using KeyedDeserializationSchema, I can get access to the partition,
offset, key and value information.
From: John O
Sent: Monday, August 20, 2018 3:15 PM
To: user
Subject: Access to kafka partition per record
I am consuming data from a kafka topic that has multiple partitions. I
I have a flink job with a big enough state that makes checkpointing long ( ~ 70
seconds).
I have configured the checkpoint timeout to 180 seconds
(setCheckpointTimeout(18))
But as you can see from the following logs, timeout seems to be ~60 seconds.
Is there another timeout configuration I n
