Re: Yarn Kerberos issue

2020-01-13 Thread Juan Gentile
-renewal-renewers-and-yarn If you think that you need more information about our issue, we can organize a call and discuss about it. Regards, Juan From: Rong Rong Date: Sunday, January 12, 2020 at 6:13 PM To: Juan Gentile Cc: Aljoscha Krettek , "user@flink.apache.org" , Arnau

Re: Yarn Kerberos issue

2020-01-10 Thread Juan Gentile
wrote: Hi, Interesting! What problem are you seeing when you don't unset that environment variable? From reading UserGroupInformation.java our code should almost work when that environment variable is set. Best, Aljoscha On 10.01.20 15:23, Juan Gentile wrote:

Re: Yarn Kerberos issue

2020-01-10 Thread Juan Gentile
erberos.login.keytab >> security.kerberos.login.principal >> security.kerberos.login.contexts >> >> >> >> Best, >> Yang >> >> Juan Gentile 于2020年1月6日周一 下午3:55写道: >> >>> Hello Rong, Che

Re: Yarn Kerberos issue

2020-01-05 Thread Juan Gentile
just not supported by Flink or I’m doing something wrong. Thank you, Juan From: Rong Rong Date: Saturday, January 4, 2020 at 6:06 PM To: Chesnay Schepler Cc: Juan Gentile , "user@flink.apache.org" , Oleksandr Nitavskyi Subject: Re: Yarn Kerberos issue Hi Juan, Chesnay was right.

Yarn Kerberos issue

2020-01-02 Thread Juan Gentile
Hello, Im trying to submit a job (batch worcount) to a Yarn cluster. I’m trying to use delegation tokens and I’m getting the following error: org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster at org.apache.flink.yarn.AbstractYarn

Problem with Flink on Yarn

2019-08-23 Thread Juan Gentile
Hello! We are running Flink on Yarn and we are currently getting the following error: 2019-08-23 06:11:01,534 WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as: (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.se

Re: Watermarks and Kafka

2019-07-08 Thread Juan Gentile
the watermarks handled in the source operator. Please let us know your opinion. Thank you, Juan G. From: Konstantin Knauf Date: Sunday, July 7, 2019 at 10:14 PM To: Juan Gentile Cc: "user@flink.apache.org" , Olivier Solliec , Oleksandr Nitavskyi Subject: Re: Watermarks and Kafka H

FlinkKafkaConsumer - Save watermark assigner in the state

2019-07-05 Thread Juan Gentile
Hello, We are currently facing an issue where we need to store the instance of the watermark and timestamp assigner in the state while consumer from Kafka. For that purpose we took a look at FlinkKafkaConsumerBase and noticed that since the methods (snapshotState and initializeState from the Ch

Watermarks and Kafka

2019-07-03 Thread Juan Gentile
Hello! We currently have a job which reads from Kafka and uses punctuated watermarks based on the messages we read. We currently keep track of the watermarks for each partition to emit a consensus watermark, taking the smallest of all partitions. We ran into an issue because we are not storing

Flink on mesos - One cluster per job - Version 1.6

2019-05-21 Thread Juan Gentile
Hello! We are trying to run in Mesos a job which will launch its own cluster (as opposed to launching the cluster and then submitting jobs to it) We have a couple of questions/issues: 1. Is there any easier way to achieve this without having to generate a graph file before submitting the jo

Flink on Mesos

2019-04-05 Thread Juan Gentile
Hello! We are having a small problem while trying to deploy Flink on Mesos using marathon. In our set up of Mesos we are required to specify the amount of disk space we want to have for the applications we deploy there. The current default value in Flink is 0 and it’s currently is not parameter

1.6 UI issues

2018-10-31 Thread Juan Gentile
Hello! We are migrating the the last 1.6 version and all the jobs seem to work fine, but when we check individual jobs through the web interface we encounter the issue that after clicking on a job, either it takes too long to load the information of the job or it never loads at all. Has anyone

Error migrating to 1.6

2018-10-24 Thread Juan Gentile
Hello! We are trying to migrate from 1.4 to 1.6 and we are getting the following exception in our jobs: org.apache.flink.util.FlinkException: The assigned slot container_e293_1539164595645_3455869_01_011241_2 was removed. at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManag

Weird behaviour after change sources in a job.

2018-09-12 Thread Juan Gentile
Hello! We have found a weird issue while replacing the source in one of our Flink SQL Jobs. We have a job which was reading from a Kafka topic (with externalize checkpoints) and we needed to change the topic while keeping the same logic for the job/SQL. After we restarted the job, instead of c

State in the Scala DataStream API

2018-08-08 Thread Juan Gentile
Hello, I'm looking at the following page of the documentation https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html particularly at this piece of code: val stream: DataStream[(String, Int)] = ... val counts: DataStream[(String, Int)] = stream .keyBy(_._1) .m

Late events in streaming using SQL API

2018-08-02 Thread Juan Gentile
Hello, We are using the SQL api and we were wondering if it’s possible to capture and log late events. We could not find a way considering the time window is managed inside the SQL. Is there a way to do this? Thank you, Juan

Process with guava cache

2018-07-31 Thread Juan Gentile
Hello! I’m trying to have a process with a cache (using guava) and following this https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/operators/process_function.html But when I run it I get the following exception: com.esotericsoftware.kryo.KryoException: java.lang.NullPointe

Re: Externalized checkpoints and metadata

2018-04-26 Thread Juan Gentile
at 20:25 To: Juan Gentile Cc: "user@flink.apache.org" , Oleksandr Nitavskyi Subject: Re: Externalized checkpoints and metadata Hi Juan, We modified the flink code a little bit to change the flink checkpoint structure so we can easily identify which is which you can read my note or th

Externalized checkpoints and metadata

2018-04-25 Thread Juan Gentile
Hello, We are trying to use externalized checkpoints, using RocksDB on Hadoop hdfs. We would like to know what is the proper way to resume from a saved checkpoint as we are currently running many jobs in the same flink cluster. The problem is that when we want to restart the jobs and pass the me

SQL Table API: Naming operations done in query

2018-02-27 Thread Juan Gentile
Hello, We are currently testing the SQL API using 1.4.0 version of Flink and we would like to know if it’s possible to name a query or parts of it so we can easily recognize what it’s doing when we run it. An additional question is, In case of small changes done to the query/ies, and assuming w