If I remember correctly, there's a fix for this in Flink 1.14 (but the
feature is disabled by default in 1.14, and enabled by default in 1.15).
(I'm thinking
that execution.checkpointing.checkpoints-after-tasks-finish.enabled [1]
takes care of this.)
With Flink 1.13 I believe you'll have to handle
With the parallelism set to 2, you will get 2 files. More than 2, actually
-- e.g., with hourly buckets, you'll get 2 files per hour.
If the i/o bandwidth of a single instance is sufficient to handle your
expected throughput, then you can set the parallelism of the sink (or the
entire pipeline) to
Doesn't it depends on 'sink.parallelism'?
If I set 'sink.parallelism' = '2' I get two files, 'sink.parallelism' = '1'
just one file...
But I think doing like that I reduce the number of tasks so it will have
negative impact on performance :-(
Sent: Tuesday, August 30, 2022 at 3:22 PM
Fro
Hi, Liu.
It seems that you may use other own jars and thay has the common-lang3 with
other versions, which may cause the version conflict.
My suggesstion is that you can shade this dependency in your own jars or in
'flink-table-planner', and the latter may require you to compile flink manually.
I am using Flink 1.13.1 on AWS EMR 6.4. I have an existing application
using DataStream API that I would like to modify to write output to S3. I
am testing the StreamingFileSink with a bounded input. I have enabled
checkpointing.
A couple questions:
1) When the program finishes, all the files r
+dev
> On Aug 30, 2022, at 11:20 AM, Rion Williams wrote:
>
>
> Hi all,
>
> I wasn't sure if this would be the best audience, if not, please advise if
> you know of a better place to ask it. I figured that at least some folks here
> either work for Ververica or might have used their platfor
How do you use `flink-sql-connector-hive-2.2.0_2.11-1.14.4.jar`? Do you use sql
client ? Do you put it in FLINK_HOME/lib?
If it's for sql client, I think you can remove the jar from FLINK_HOME/lib, but
add it in Flink SQL client using `add jar
'flink-sql-connector-hive-2.2.0_2.11-1.14.4.jar' `,
Hi, i got NoSuchMethodError when using flink
flink-sql-connector-hive-2.2.0_2.11-1.14.4.jar.
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Unexpected exception. This is a bug. Please consider filing an issue.
at
org.apache.flink.table.client.SqlClient.start
