Re: How to mount PVC volumes using Flink Native Kubernetes ?

Thanks Tao for providing your internal use case. I have create a ticket for this feature[1]. [1]. https://issues.apache.org/jira/browse/FLINK-24332 Best, Yang tao xiao 于2021年9月11日周六 上午10:18写道： > Thanks David for the tips. We have been running Flink with no performance > degradation observed i

Re: RocksDB state not cleaned up

Thanks for the feedback! However TTL already proves that the state cannot be cleaned up on time due to too many levels built up in RocksDB. Hi @Yun Tang do you have any suggestions to tune RocksDB to accelerate the compaction progress? On Fri, Sep 17, 2021 at 8:01 PM David Morávek wrote: > Cle

Re: Built-in functions to manipulate MULTISET type

Hi Kai, AFAIK, there is no built-in function to extract the keys in MULTISET to be an ARRAY. Define a UTF is a good solution. Best, JING ZHANG Kai Fu 于2021年9月18日周六 上午7:35写道： > Hi team, > > We want to know if th

Re: Invalid flink-config keeps going and ignores bad config values

AFAIK there is not. Flink will just skip the invalid lines. Best, Yangze Guo On Sat, Sep 18, 2021 at 7:00 AM Dan Hill wrote: > > Hi. I noticed my flink-config.yaml had an error in it. I assumed a bad > config would stop Flink from running (to catch errors earlier). Is there a > way I can en

Built-in functions to manipulate MULTISET type

Hi team, We want to know if there is any built-in function to extract the keys in MULTISET to be an ARRAY. There is no such function as far as we can find, except to define a simple wrapper UDF for that, please ad

Invalid flink-config keeps going and ignores bad config values

Hi. I noticed my flink-config.yaml had an error in it. I assumed a bad config would stop Flink from running (to catch errors earlier). Is there a way I can enable a strict parsing mode so any Flink parsing issue causes Flink to fail? I don't see one when looking at the code. 2021-09-17 22:45:1

RE: hdfs lease issues on flink retry

Hi Matthias, Thanks for looking into the issue and creating a ticket. I am thinking of having a workaround until the issue is fixed. What if I create the attempt directories with a random int by patching HadoopOutputFormatBase’s open() method? Original: TaskAttemptID taskAttemptID = TaskAttem

Re: Building a flink connector

Hi Lars, We're actually working on creating a guide to help our users on how to create a connector. Perhaps it would be good to use your needs to see what we need to include in such a guide, so we can make that available to the community. Would be great if we can have a conversation on that topic.

Re: Building a flink connector

Hi Lars, We've built a custom connector for Snowflake (both source and sink). Feel free to reach out in private if you have any questions. On Fri, Sep 17, 2021, 14:33 Lars Skjærven wrote: > We're in need of a Google Bigtable flink connector. Do you have any tips > on how this could be done, e.

Re: Cleaning old incremental checkpoint files

Hi Robin, You could use Checkpoints#loadCheckpointMetadata[1] to analysis the checkpoint meta data. For the problem of make checkpoint self-contained, you might be interested in the ticket [2] [1] https://github.com/apache/flink/blob/8debdd06be0e917610c50a77893f7ade45cee98f/flink-runtime/src

Re: Flink S3A failed to connect to service endpoint from IntelliJ IDE

Hi James, For one thing, It looks to me that we should not configure the credential in pom.xml, instead, we might introduce a core-site.xml under the classpath and configured like fs.s3a.access.key fs.s3a.secret.key I tried with the abov

Re: RocksDB state not cleaned up

Cleaning up with timers should solve this. Both approaches have some advantages and disadvantages though. Timers: - No "side effects". - Can be set in event time. Deletes are regular tombstones that will get compacted later on. TTL: - Performance. This costs literally nothing compared to an extra

Exception by flink kafka

HI, In what scenarios we hit with *java.lang.OutOfMemoryError: Java heap space while publishing to kafka . I hit with this exception and a resolution added property *.setProperty("security.protocol","SSL");in the flink application. Later I started encountering org.apache.kafka.common.errors.Timeou

Building a flink connector

We're in need of a Google Bigtable flink connector. Do you have any tips on how this could be done, e.g. general guidelines on how to write a connector ? Thanks, Lars

Re: Re: Savepoints with bootstraping a datastream function

Hi Rakshit, I think FLIP-147 might still not be able to support this case, since for bounded jobs, it supports each task exit after a checkpoint to commit the remaining data, but it could not ensures all the tasks exit after the same checkpoint; for savepoint, it could not supporting taking a sav

Re: Savepoints with bootstraping a datastream function

Hi Arvid. I went through the code, confluence and jira on FLIP-147. I couldn't determine if it's possible to manually trigger a savepoint/checkpoint as I couldn't find any javadoc apis for the same. Also, would the setting "ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH" still create a checkpoint if my enti