Connecting to kinesis with mfa

Hi guys, we are struggling to connect to kinesis when mfa is activated. I did configured everything according to the documentation but still getting exception : val producerConfig = new Properties() producerConfig.put(AWSConfigConstants.AWS_REGION, awsRegion) producerConfig.put(AWSConfigConstants

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Thanks for the thorough update! I'll answer inline. On 14.12.20 16:33, Yun Gao wrote: 1. To include EndOfPartition into consideration for barrier alignment at the TM side, we now tend to decouple the logic for EndOfPartition with the normal alignment behaviors to avoid the complex interfe

Re: FlinkSQL kafka->dedup->kafka

HI Laurent, Did you manage to find the error in your MATCH_RECOGNIZE statement? If I had to take a guess, I'd say it's because you are accessing A, but due to the quantifier of * there might actually be no event A. Cheers, Konstantin On Fri, Nov 27, 2020 at 10:03 PM Laurent Exsteens < laurent

Flink 1.12 and Stateful Functions

Hi, just a quick question: Is there a rough estimation, when the Flink 1.12 Features (especially the new HA-Mode) will also be available in Flink Stateful Functions? Best regards Jan

Re: Flink 1.12

Thanks. Do you have ETA for docker images? > On Dec 14, 2020, at 3:43 AM, Chesnay Schepler wrote: > > 1) It is compiled with Java 8 but runs on Java 8 & 11. > 2) Docker images are not yet published. > 3) It is mentioned at the top of the Kubernetes HA Services documentation > that it also work

Re: Flink 1.12

Unfortunately no; there are some discussions going on in the docker-library/official-images PR that have to be resolved first, but currently these would require changes on the Flink side that we cannot do (because it is already relea

Flink - Create Temporary View and "Rowtime attributes must not be in the input rows of a regular join"

When I try to refactor my joins into a temporary view to share joins and state, I get the following error. I tried a few variations of the code snippets below (adding TIMESTAMP casts based on Google searches). I removed a bunch of fields to simplify this example. Is this a known issue? Do I hav

Is working with states supported in pyflink1.12?

Hello, I'm new to flink and trying to build a stateful application using python datastream API but can't find any example of how to use states in python in flink 1.12 documentation. Is states supported in the python datastream API?And if so, how can I use it? Thanks in advance!

Re: Is working with states supported in pyflink1.12?

It is currently not possible to access state with the Python API. A proposal has recently been made to enhance the API with state access (under FLIP-152), but at this time I cannot provide a prediction for when it might be released. On 12/15/2020 7:55 PM, Nadia Mostafa wrote: Hello, I'm new

Re: Flink 1.12

Thanks Chesney for your quick response, I read documentation https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink#FLIP144:NativeKubernetesHAforFlink-NativeK8s

Re: Is working with states supported in pyflink1.12?

Hi, As Chesnay said, PyFlink has already supported Python DataStream stateless APIs so that users are able to perform some basic data transformations, but doesn't provide state access support yet in release-1.12. The proposal[1] of enhancing the API with state access has been made and related disc

Re: Flink 1.12

Hi Boris, What is -p 10? It is same to --parallelism 10. Set the default parallelism to 10. does it require a special container build? No, the official flink docker image could be used directly. Unfortunately, we do not have the image now. And we are trying to figure out. You could follow the i

Flink - sending clicks+impressions to AWS Personalize

I want to try using AWS Personalize to get content recommendations. One of the fields on the input (click) event is a list of recent impressions. E.g. { ... eventType: 'click', eventId: 'click-1', itemId: 'item-1' impression: ['item-2', 'item-3', 'i

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Hi Aljoscha, Very thanks for the feedbacks! For the remaining issues: > 1. You mean we would insert "artificial" barriers for barrier 2 in case we receive EndOfPartition while other inputs have already received barrier 2? I think that makes sense, yes. Yes, exactly, I

Get current kafka offsets for kafka sources

Hi, Is there any way to fetch the current kafka topic offsets for the kafka sources for flink? Thanks! -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com | BLOG | FOLLOW US | LIKE US

Re: Get current kafka offsets for kafka sources

My understanding is the FlinkKafkaConsumer is a wrapper around the Kafka consumer libraries so if you've set the group.id property you should be able to see the offsets with something like kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-flink-application. On Tue, D

Re: Get current kafka offsets for kafka sources

I'll give a look into that approach. Thanks On Tue, Dec 15, 2020 at 9:48 PM Aeden Jameson wrote: > My understanding is the FlinkKafkaConsumer is a wrapper around the > Kafka consumer libraries so if you've set the group.id property you > should be able to see the offsets with something like > ka

Re: Connecting to kinesis with mfa

Hey Avi, Maybe providing secret/access key + session token doesn't work, and you need to provide either one of them? https://docs.aws.amazon.com/credref/latest/refdocs/setting-global-aws_session_token.html I'll also ping some AWS contributors active in Flink to take a look at this. Best, Robert

Re: Direct Memory full

Hey Rex, the direct memory is used for IO. There is no concept of direct memory being "full". The only thing that can happen is that you have something in place (Kubernetes, YARN) that limits / enforces the memory use of a Flink process, and you run out of your memory allowance. The direct memory

Re: pause and resume flink stream job based on certain condition

What you can also do is rely on Flink's backpressure mechanism: If the map operator that validates the messages detects that the external system is down, it blocks until the system is up again. This effectively causes the whole streaming job to pause: the Kafka source won't read new messages. On T

Re: Get current kafka offsets for kafka sources

Hey, Have a look at [1]. Basically, you won't see the "real-time" consumer group offsets stored in Kafka itself, but only the ones the Flink Kafka consumer stores there when checkpointing (assuming you have checkpointing enabled). The same information is available in Flink metrics [2], "committedO

failed w/ Application Mode but succeeded w/ Per-Job Cluster Mode

Hi, I have an artifact which works perfectly fine with Per-Job Cluster Mode with the following bash script: #!/bin/env bash export FLINK_CONF_DIR=./conf export HADOOP_CLASSPATH=`hadoop classpath` $FLINK_HOME/bin/flink run -t yarn-per-job myjar.jar myconf.conf I tried Application Mode [1] usi

Re: failed w/ Application Mode but succeeded w/ Per-Job Cluster Mode

I just added the following option to the script: -Dclassloader.parent-first-patterns.additional=org.apache.kafka.common.serialization Now it seems to work. Why do the application mode and the per-job cluster mode behave differently when it comes to the classloading? Is it a bug? or intended? B

Re: Flink jobmanager TLS connectivity to Zookeeper

Hey Azeem, I haven't tried this myself, but from the code / documentation, this could work: Flink ships with ZK 3.4 by default. You need to remove the ZK3.4 jar file from the lib/ folder and add the ZK3.5 file from opt/ to lib/. According to this guide, you could try passing the SSL configuratio