Hello,
I am frequently seeing this error in my jobmanager logs:
2019-11-18 09:07:08,863 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
kdi_g2 -> (Sink: s_id_metadata, Sink: t_id_metadata) (23/24)
(5e10e88814fe4ab0be5f554ec59bd93d) switched from RUNNING to FAILED.
jav
Is it possible to configure certain settings at runtime, on a per-job basis
rather than globally within flink-conf.yaml?
For example, I have a job where it's desirable to retain a large number of
checkpoints via
state.checkpoints.num-retained.
The checkpoints are cheap, and it's low cost. For oth
The Flink Kafka consumer's watermark extractor claims:.
"The watermark extractor will run per Kafka partition, watermarks will be
merged across partitions in the same way as in the Flink runtime, when
streams are merged."
Suppose we have simple situation where two streams merge into one. Is the
o
Hello,
Exact semantics for checkpointing/task recovery are still a little
confusing to me after parsing docs: so a few questions.
- What does Flink consider a task failure? Is it any exception that the job
does not handle?
- Do the failure recovery strategies mentioned in
https://ci.apache.org/pr
Hello,
Suppose I am using a *nondeterministic* time based partitioning scheme (e.g
Flink processing time) to bucket S3 objects via the *BucketAssigner*,
designated using *BulkFormatBuilder* for StreamingFileSink.
Suppose that after an S3 MPU has completed, but *before* Flink internally
commits (w
Hello,
I am using StreamingFileSink, KafkaConsumer010 as a Kafka -> S3 connector
(Flink 1.8.1, Kafka 0.10.1).
The setup is simple:
Data is written first bucketed by datetime (granularity of 1 day), then by
kafka partition.
I am using *event time* (Kafka timestamp, recorded at the time of creation
);
> }
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/serialization/BulkWriter.html#finish--
>
> Regards,
> Ravi
>
> On Tue, Oct 22, 2019 at 4:10 AM amran dean wrote:
>
>> Hello,
>> I'm using BulkWri
Hello,
I'm using BulkWriter to write newline-delimited, LZO-compressed files. The
logic is very straightforward (See code below).
I am experiencing an issue decompressing the created files created in this
manner, consistently getting "lzop: unexpected end of file". Is this an
issue with caller of
Hello,
StreamingFileSink's part file naming convention is not adjustable. It has
form: *part--. *
My use case for StreamingFileSink is a Kafka -> S3 pipeline, and files are
read and processed from S3 using spark. In almost all cases, I want to
compress raw data before writing to S3 using the BulkF
Hello,
I am writing a custom S3 object encoder (code here:
https://pastebin.com/raw/9Ag4ZVpX) used via:
StreamingFileSink dataSink = StreamingFileSink
.forRowFormat(new Path("s3a://some_path"), new RecordDataSerializer())
During execution, it does not produce any data in S3. This pipeline
I am evaluating StreamingFileSink (Kafka 0.10.11) as a production-ready
alternative to a current Kafka -> S3 solution.
Is there any way to verify the integrity of data written in S3? I'm
confused how the file names (e.g part-1-17) map to Kafka partitions, and
further unsure how to ensure that no K
11 matches
Mail list logo
