Handling stale data enrichment

do we know if it is stale data or not based on timestamp (watermark) as it can happen that a particular enriched record is not updated for 6 hrs. Regards, Vinay Patil

Re: Handling stale data enrichment

o handle late record). I think the downstream consumer of enriched data will have to dedup the duplicate records or else we will end up having stale enrichment. Regards, Vinay Patil On Fri, Apr 24, 2020 at 12:14 PM Konstantin Knauf wrote: > Hi Vinay, > > I assume your subscription up

Timer metric in Flink

also have to create Timer interface and add it to the metric group. Is this possible? I want to have a timer to check Hbase lookup time. Regards, Vinay Patil

Re: Timer metric in Flink

Ohh Okay, basically implement the Gauge and add timer functionality to it for now. Is there a plan or JIRA ticket to add Timer metric in future release, I think it is good to have Regards, Vinay Patil On Wed, Jun 10, 2020 at 5:55 PM Chesnay Schepler wrote: > You cannot add custom met

Flink Deployment on Kubernetes session Cluster

Regards, Vinay Patil

Re: Flink Deployment on Kubernetes session Cluster

Thanks and Regards, Vinay Patil On Wed, Jul 29, 2020 at 11:08 AM Yang Wang wrote: > Hi Vinay Patil, > > You are right. Flink does not provide any isolation between different jobs > in the same Flink session cluster. > You could use Flink job cluster or application cluster(from 1.11)

Flink InfluxDb connector not present in Maven

Hi Team, Flink influx db connector `flink-connector-influxdb_2.1` is not present in Maven , can you please upload the same https://repo.maven.apache.org/maven2/org/apache/bahir/ Regards, Vinay Patil

Re: Consuming data from dynamoDB streams to flink

the consumer is lagging behind. Regards, Vinay Patil On Fri, Jul 19, 2019 at 10:40 PM Andrey Zagrebin wrote: > Hi Vinay, > > 1. I would assume it works similar to kinesis connector (correct me if > wrong, people who actually developed it) > 2. If you have activated just che

Re: Consuming data from dynamoDB streams to flink

Hello, For anyone looking for setting up alerts for flink application ,here is good blog by Flink itself : https://www.ververica.com/blog/monitoring-apache-flink-applications-101 So, for dynamoDb streams we can set the alert on millisBehindLatest Regards, Vinay Patil On Wed, Aug 7, 2019 at 2

Using STSAssumeRoleSessionCredentialsProvider for cross account access

at the credentials are not required to be passed : https://github.com/apache/flink/blob/abbd6b02d743486f3c0c1336139dd6b3edd20840/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/AWSUtil.java#L164 Regards, Vinay Patil

Re: Using STSAssumeRoleSessionCredentialsProvider for cross account access

Hi, Can someone pls help here , facing issues in Prod . I see the following ticket in unresolved state. https://issues.apache.org/jira/browse/FLINK-8417 Regards, Vinay Patil On Thu, Oct 24, 2019 at 11:01 AM Vinay Patil wrote: > Hi, > > I am trying to access dynamo streams from a

DynamoStreams Consumer millisBehindLatest metric

supports DynamoStreams Regards, Vinay Patil

Re: Using STSAssumeRoleSessionCredentialsProvider for cross account access

Thanks Fabian, @Gordon - Can you please help here. Regards, Vinay Patil On Fri, Oct 25, 2019 at 9:11 PM Fabian Hueske wrote: > Hi Vinay, > > Maybe Gordon (in CC) has an idea about this issue. > > Best, Fabian > > Am Do., 24. Okt. 2019 um 14:50 Uhr schrieb Vinay

Streaming File Sink - Parquet File Writer

RowFormatBuilder. P.S. Curious to know Why was the RollingPolicy not exposed in case of BulkFormat ? Regards, Vinay Patil

Strictly use TLSv1.2

ink@taskmanager1:port/user/taskmanager) Now, when I hit the above command for the data port, it does not allow TLSv1.1 and only allows TLSv1.2 Can you please let me know how can I enforce all the flink ports to use TLSv1.2. Regards, Vinay Patil -- Sent from: http://apache-flink-user-mailing-l

Re: Strictly use TLSv1.2

Hi, Can someone please help me with this issue. Regards, Vinay Patil -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Strictly use TLSv1.2

ink@taskmanager1:port/user/taskmanager) Now, when I hit the above command for the data port, it does not allow TLSv1.1 and only allows TLSv1.2 Can you please let me know how can I enforce all the flink ports to use TLSv1.2. Regards, Vinay Patil

Re: Strictly use TLSv1.2

Hi Fabian, Created a JIRA ticket : https://issues.apache.org/jira/browse/FLINK-9643 Regards, Vinay Patil On Fri, Jun 22, 2018 at 1:25 PM Fabian Hueske wrote: > Hi Vinay, > > This looks like a bug. > Would you mind creating a Jira ticket [1] for this issue? > > Thank you v

Query regarding rest.port property

Web UI running or am I missing any configuration ? Regards, Vinay Patil

Re: Query regarding rest.port property

still not able to hit the rest api's, Is there anything else I can do here ? Yes, you are right about separating the API's into two parts. Regards, Vinay Patil On Sat, Jul 21, 2018 at 1:46 AM Chesnay Schepler wrote: > Something that I was thinking about a while ago was to separate t

Checkpointing not happening in Standalone HA mode

do not see any checkpoints triggered on Flink UI. Am I missing any configurations to be set for the RemoteExecutionEnvironment for checkpointing to work. Regards, Vinay Patil

Re: Checkpointing not happening in Standalone HA mode

No error in the logs. That is why I am not able to understand why checkpoints are not getting triggered. Regards, Vinay Patil On Wed, Jul 25, 2018 at 4:44 PM Vinay Patil wrote: > Hi Chesnay, > > No error in the logs. That is why I am not able to understand why > checkpoints

Re: Checkpointing not happening in Standalone HA mode

Hi Chesnay, No error in the logs. That is why I am not able to understand why checkpoints are getting triggered. Regards, Vinay Patil On Wed, Jul 25, 2018 at 4:36 PM Chesnay Schepler wrote: > Please check the job- and taskmanager logs for anything suspicious. > > On 25.07.2018 12:

Re: Checkpointing not happening in Standalone HA mode

Source is not being executed at the moment. Aborting checkpoint. In the pipeline I have a stream initialized using "fromCollection". I think I will have to get rid of this. What do you suggest Regards, Vinay Patil On Thu, Jul 26, 2018 at 12:04 PM vino yang wrote: > Hi Vinay: > >

Re: Checkpointing not happening in Standalone HA mode

Hi Vino, Yes, Job runs successfully, however, no checkpoints are successful. I will update the source Regards, Vinay Patil On Fri, Jul 27, 2018 at 2:00 PM vino yang wrote: > Hi Vinay, > > Oh! You use a collection source? That's the problem. Please use a general > source lik

Unable to start session cluster using Docker

lhost in /etc/hosts file. Can you please let me know what is the issue here. Regards, Vinay Patil

Re: Unable to start session cluster using Docker

Thank you Till, I am able to start the session-cluster now. Regards, Vinay Patil On Fri, Oct 5, 2018 at 8:15 PM Till Rohrmann wrote: > Hi Vinay, > > are you referring to flink-contrib/docker-flink/docker-compose.yml? We > recently fixed the command line parsing with Flink 1.5.4 an

Re: AvroInputFormat Serialisation Issue

a:93) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) Please let me know if there is a fix for this issue as I have not faced this problem for DataStreams. Regards, Vinay Patil -- Sent fro

Re: AvroInputFormat Serialisation Issue

Hi, Changing the classloader config to parent-first solved the issue. Regards, Vinay Patil On Wed, Nov 7, 2018 at 7:25 AM Vinay Patil wrote: > Hi, > > Can someone please help here. > > On Nov 6, 2018 10:46 PM, "Vinay Patil [via Apache Flink User Mailing List > arc

Re: StreamingFileSink cannot get AWS S3 credentials

Hi, Can someone please help on this issue. We have even tried to set fs.s3a.impl in core-site.xml, still its not working. Regards, Vinay Patil On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User Mailing List archive.] wrote: > Hi All, > We have implemented S

Re: StreamingFileSink cannot get AWS S3 credentials

. We tried adding fs.s3a.impl to core-site.xml when the default configurations were not working. Regards, Vinay Patil On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann wrote: > Hi Vinay, > > Flink's file systems are self contained and won't respect the > core-site.xml if I'

Re: StreamingFileSink cannot get AWS S3 credentials

Hi Till, Can you please let us know the configurations that we need to set for Profile based credential provider in flink-conf.yaml Exporting AWS_PROFILE property on EMR did not work. Regards, Vinay Patil On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann wrote: > The old BucketingSink was us

Re: EXT :Re: StreamingFileSink cannot get AWS S3 credentials

Hi Stephan., Yes, we tried setting fs.s3a.aws.credentials.provider but we are getting class not found exception for InstanceProfileCredentialsProvider because of shading issue. Regards, Vinay Patil On Thu, Jan 17, 2019 at 3:02 PM Stephan Ewen wrote: > Regarding configurations: According

Re: Consuming data from dynamoDB streams to flink

recommended parallelism to be set for the source , should it be one to one mapping , for example if there are 3 shards , then parallelism should be 3 ? Regards, Vinay Patil On Wed, Aug 1, 2018 at 3:42 PM Ying Xu [via Apache Flink Mailing List archive.] wrote: > Thank you so much Fabian! >

StackOverflow Error

ms-dynamo-streams", new JsonSerializationSchema())) .name("Kafka Sink"); try { env.execute(); } catch (Exception e) { System.out.println("Caught exception for pipeline" + e.getMessage()); e.printStackTrace(); } } Regards, Vinay Patil

Re: StackOverflow Error

Hi Vinay, > > Please make sure that all your custom code is serializable. You can run > this using new mode. > > Thanks, > Ravi > > On Sat 20 Jul, 2019, 08:13 Vinay Patil, wrote: > >> Hi, >> >> I am trying to run a pipel

Re: StackOverflow Error

Hi Ravi, The uber jar was correct, adding ClosureCleanerLevel to TOP_LEVEL resolved this issue. Thanks a lot. Is there any disadvantage of explicitly setting this ? Regards, Vinay Patil On Sat, Jul 20, 2019 at 10:23 PM Ravi Bhushan Ratnakar < ravibhushanratna...@gmail.com> wrote: >

[Discuss] Ordering of Records

the same TaskManager slot, if possible. Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Discuss-Ordering-of-Records-tp7933.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: [Discuss] Ordering of Records

Hi Robert, This was the same posted by me in stack overflow since I was not getting any reply here :) Regards, Vinay Patil On Thu, Jul 14, 2016 at 6:55 PM, rmetzger0 [via Apache Flink User Mailing List archive.] wrote: > There is a parallel thread answering the questions going on here alre

Regarding QueryableState

. Thanks, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-QueryableState-tp8324.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Facing Issues while migrating to S3A

Hi All, I am facing issues when I changed from S3n to S3A, I am using hadoop-aws-2.7.2 and aws-java-sdk-1.7.4 as dependencies Getting this error : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access Tried to change the fs.s3.buffer.dir path but getting same error

Re: Regarding QueryableState

Thanks Ufuk, will take a look at it. Regards, Vinay Patil On Thu, Aug 4, 2016 at 4:26 PM, Ufuk Celebi [via Apache Flink User Mailing List archive.] wrote: > You can expect it to be merged by the end of this week. > > Note that the APIs are very low level at the moment though.

Re: Facing Issues while migrating to S3A

p.dll"* I want to test it locally so that I don't have to run every-time on cluster Regards, Vinay Patil On Fri, Aug 5, 2016 at 3:18 PM, Stefan Richter [via Apache Flink User Mailing List archive.] wrote: > Hi, > > from the exception, it seems like some native library is mis

Dealing with Multiple sinks in Flink

issue. (running from IDE is resulting in same output, Tested this with Flink 1.0.3 and 1.1.1) I am not understanding why this is happening. Regards, Vinay Patil

Re: Dealing with Multiple sinks in Flink

Hi Max, I tried writing to local file as well, its giving me the same issue, I have attached the logs and dummy pipeline code. logs.txt dummy_pipeline.txt

Re: Dealing with Multiple sinks in Flink

Hi, Just an update, the window is not getting triggered when I change the parallelism to more than 1. Can you please explain why this is happening ? Regards, Vinay Patil On Wed, Aug 24, 2016 at 9:55 AM, vinay patil [via Apache Flink User Mailing List archive.] wrote: > Hi Max, > >

Re: Dealing with Multiple sinks in Flink

Hi Max, Here is the code for Timestamp assigner and watermark generation. PFA Regards, Vinay Patil On Thu, Aug 25, 2016 at 7:39 AM, Maximilian Michels [via Apache Flink User Mailing List archive.] wrote: > I'm assuming there is something wrong with your Watermark/Timestamp > assi

Re: Dealing with Multiple sinks in Flink

AscendingTimeStampExtractor but facing the same issue. Regards, Vinay Patil On Thu, Aug 25, 2016 at 1:32 PM, rmetzger0 [via Apache Flink User Mailing List archive.] wrote: > Flink 1.1.1 has a metric for exposing the low watermark of each operator. > Maybe you can access a TaskManager via JMX to see the value

Enabling Encryption between slaves in Flink

/flink-user/201601.mbox/%3ccanc1h_v0pqfvfto478ft5cbgm-bf-do742gouz528bw7vrj...@mail.gmail.com%3E Regards, Vinay Patil

Data Transfer between TM should be encrypted

ta at the start of each operator and encrypt it at the end of each operator, but I feel this is not an efficient approach. I just want to check if there are alternatives to this and can this be achieved by doing some configurations. Regards, Vinay Patil -- View this message in context:

Re: Data Transfer between TM should be encrypted

Hi Stephan, Thank you for your reply. Till when can I expect this feature to be integrated in master or release version ? We are going to get production data (financial data) in October end , so want to have this feature before that. Regards, Vinay Patil On Mon, Aug 29, 2016 at 11:15 AM

Re: Data Transfer between TM should be encrypted

Hi Vijay, That's a good news for me. Eagerly waiting for this change so that I can integrate and test it before going live. Regards, Vinay Patil On Tue, Aug 30, 2016 at 4:06 PM, Vijay Srinivasaraghavan [via Apache Flink User Mailing List archive.] wrote: > Hi Stephan, > > T

Re: Streaming - memory management

since I read the above part in documentation so we are going for Cassandra now (to store records and query them for a special case) Regards, Vinay Patil On Wed, Aug 31, 2016 at 4:51 AM, Stephan Ewen wrote: > In streaming, memory is mainly needed for state (key/value state). The >

Re: Streaming - memory management

data. Regards, Vinay Patil On Wed, Aug 31, 2016 at 11:34 AM, Fabian Hueske wrote: > Hi Vinaj, > > if you use user-defined state, you have to manually clear it. > Otherwise, it will stay in the state backend (heap or RocksDB) until the > job goes down (planned or due to an OOM erro

Re: Streaming - memory management

Hi Fabian/Stephan, Waiting for your suggestion Regards, Vinay Patil On Wed, Aug 31, 2016 at 1:46 PM, Vinay Patil wrote: > Hi Fabian/Stephan, > > This makes things clear. > > This is the use case I have : > I am performing a outer join operation on the two streams (in windo

Re: Streaming - memory management

using Flink Connector, but I think this will be a better option rather than hitting the query to external store since RocksDb will store the data to local TM disk, the retrieval will be faster here than Cassandra , right ? What do you think ? Regards, Vinay Patil On Thu, Sep 1, 2016 at 10:19 AM, F

Re: Streaming - memory management

g and flatmap to take care of rest logic. Or are you suggestion to use Co-FlatMapFunction after the outer-join operation (I mean after doing the window and getting matchingAndNonMatching stream )? Regards, Vinay Patil On Thu, Sep 1, 2016 at 11:38 AM, Fabian Hueske-2 [via Apache Flink User Mailing

Re: Streaming - memory management

ained for user-defined operators to avoid confusion. Regards, Vinay Patil On Thu, Sep 1, 2016 at 1:12 PM, Fabian Hueske-2 [via Apache Flink User Mailing List archive.] wrote: > I thought you would like to join the non-matched elements with another > (third) stream. > > --> s1.unio

Re: Streaming - memory management

Hi Fabian, https://ci.apache.org/projects/flink/flink-docs-master/dev/state_backends.html#the-rocksdbstatebackend I am referring to this, this does not clearly state if the state will be maintained in local disk even after checkpointing. Or I am not getting it correclty :) Regards, Vinay Patil

Re: Data Transfer between TM should be encrypted

Hi Vijay, Did you raise the PR for this task, I don't mind testing it out as well. Regards, Vinay Patil On Tue, Aug 30, 2016 at 6:28 PM, Vinay Patil wrote: > Hi Vijay, > > That's a good news for me. Eagerly waiting for this change so that I can > integrate and tes

Thread Locals in Flink Streaming Application

this ? Can we make use of open method to avoid ThreadLocal by initializing the variable there, since the open will be called once per thread (correct me here) Can you please explain this to make it clear Regards, Vinay Patil

Re: Flink Checkpoint runs slow for low load stream

. Regards, Vinay Patil On Mon, Sep 26, 2016 at 10:08 AM, Chakravarthy varaga [via Apache Flink User Mailing List archive.] wrote: > Hi Stefan, > > Please find my responses below. > > - What source are you using for the slow input? > * [CVP] - Both stream as point

Re: Flink Checkpoint runs slow for low load stream

I am not sure about that, I will run the pipeline on cluster and share the details Since window is a stateful operator , it will store only the key part in the state backend and not the value , right ? Regards, Vinay Patil On Mon, Sep 26, 2016 at 12:13 PM, Stephan Ewen [via Apache Flink User

Re: Flink Checkpoint runs slow for low load stream

? Regards, Vinay Patil On Tue, Sep 27, 2016 at 3:51 AM, Stephan Ewen [via Apache Flink User Mailing List archive.] wrote: > @vinay - Window operators store everything in the state backend. > > On Mon, Sep 26, 2016 at 7:34 PM, vinay patil <[hidden email] > <http:///user/SendEmail.jtp?typ

Re: Flink Checkpoint runs slow for low load stream

Thanks Stephan for your inputs We are getting the checkpointing issue for other projects as well in which the window and encryption stuff is not there (using Flink 1.1.1). As you suggested, I will try using RocksDB and run the pipeline on EMR to provide more details. Regards, Vinay Patil On

Regarding Late Elements

Aljoscha) to even accept late elements. Regards, Vinay Patil

Re: Regarding Late Elements

Hi LF, So did you manage to get the workaround for it ? I am using a Custom Trigger which is similar to 1.0.3 with few changes Regards, Vinay Patil On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.] wrote: > We have the same requirement - we cannot disc

Re: Regarding Late Elements

the change I have done in my custom trigger which is similar to 1.0.3 , does this looks good to you ? (I am not using canMerge and onMerge methods since I am not aware of it) Regards, Vinay Patil On Tue, Oct 4, 2016 at 3:58 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.] wr

Re: Regarding Late Elements

window and purge it* I have kept the onEventTime code same as 1.0.3, where it will fire and purge Am I doing something wrong ? (I have kept allowedLateness to Long.MAX_VALUE) Regards, Vinay Patil On Tue, Oct 4, 2016 at 9:58 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.] wrote:

Re: Regarding Late Elements

Merge functions in trigger, from the javadocs I got it on a high level, but not able to understand it correctly (so I am not using these functions for now in Custom Trigger). Regards, Vinay Patil On Tue, Oct 4, 2016 at 10:51 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.] wrote: >

Re: Regarding Late Elements

Yes Kostas, thank you for the explanation , I will take a look Regards, Vinay Patil On Tue, Oct 4, 2016 at 11:23 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.] wrote: > Hi Vinay, > > These methods are useful when using your trigger with SessionWindows. When > u

Re: Data Transfer between TM should be encrypted

Hi Stephan, https://github.com/apache/flink/pull/2518 Is this pull request going to be part of 1.2 release ? Just wanted to get an idea on timelines so that I can pass on to the team. Regards, Vinay Patil On Thu, Sep 15, 2016 at 11:45 AM, Vijay Srinivasaraghavan < vijikar...@yahoo.com>

Re: Data Transfer between TM should be encrypted

Hi Robert, Thank you for that information, can you please let me know when 1.2 is planned to release ? Regards, Vinay Patil On Wed, Oct 12, 2016 at 4:17 AM, rmetzger0 [via Apache Flink User Mailing List archive.] wrote: > Hi, > I think that pull request will be merged for 1.2. > >

Re: Data Transfer between TM should be encrypted

Thank you Stephan and Robert for your positive response Regards, Vinay Patil On Thu, Oct 13, 2016 at 11:22 AM, Stephan Ewen [via Apache Flink User Mailing List archive.] wrote: > I agree with Robert. We should probably aim for the release to be end of > November, so that it will come thi

Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn

let me know what I am doing wrong here. Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9702.html Sent from the Apache Flink User Mailing List archive. mailing list archive at

Kafka Monitoring

running the application. Has anyone faced this issue ? I am using Kafka 0.9.0.1 Regards, Vinay Patil

Re: Kafka Monitoring

o the kafka offset checker is not showing the consumers. Is there any configuration I am missing ? Regards, Vinay Patil On Tue, Nov 8, 2016 at 2:32 AM, Daniel Santos [via Apache Flink User Mailing List archive.] wrote: > Hello, > > I have been using that setup. > From my unde

Re: Kafka Monitoring

to documentation of 0.9.x the offsets are stored in Kafka, but we can set offset.storage property to zookeeper. Even by setting this I am not getting the consumer listed. Kafka cli command also does not show this consumer Regards, Vinay Patil On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink

Re: Kafka Monitoring

Hi Limbo, I am using 0.9, I am not able to see updated results even after refreshing. There is some property that we have to set in order to make this work Regards, Vinay Patil On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing List archive.] wrote: > I am using kafka

Re: Kafka Monitoring

to zookeeeper, but still not getting the consumers listed I am using Kafka 0.9.0.1 Regards, Vinay Patil On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User Mailing List archive.] wrote: > Hello, > > This is my config. > > On kafka props : > > va

Re: Kafka Monitoring

as brokerPath ? this is my entry in server.prop file -> zookeeper.connect=localhost:2181 Have you set this as zkhost:2181/kafka09 ? Regards, Vinay Patil On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User Mailing List archive.] wrote: > Hi, > > Your kafka broker

Re: Kafka Monitoring

Yes Kafka and Flink connect to that zookeeper only. Not sure why it is not listing the consumer Regards, Vinay Patil On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User Mailing List archive.] wrote: > Hi, > > brokerPath is just optional. > > Used if you want

Re: Kafka Monitoring

the brokerPath ? I mean for which property have you set it Regards, Vinay Patil On Wed, Nov 9, 2016 at 12:32 AM, Daniel Santos [via Apache Flink User Mailing List archive.] wrote: > Hello, > > On flink do you have the checkpoint enabled ? > > env.enableCheckpointing(interval =

Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn

Hi Robert, Sorry for the late response. I checked the TM and JM logs but am not able to see that file added in the classpath. I have deployed Flink on EMR, and placed the yaml file in /usr/lib/flink/lib folder. What am I doing wrong here ? Regards, Vinay Patil On Wed, Oct 26, 2016 at 1:25 PM

Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn

Hi Max, When I provided the config file using --yarnship option , it worked. Thank you. However I did not get this : " *it's better to add it to the ship files in the * *cluster descriptor*" Can you please tell me how to do this ? Regards, Vinay Patil On Fri, Nov 18,

CodeAnalysisMode in Flink

to analyse my pipeline to get rid of common mistakes. Regards, Vinay Patil

Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn

OK, got that, thank you for your solution, it worked for me with --yarnship option Regards, Vinay Patil On Tue, Nov 22, 2016 at 4:44 PM, Maximilian Michels [via Apache Flink User Mailing List archive.] wrote: > Hi Vinay, > > I was referring to setting up an deploying a YARN cluster

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Hi Sendoh, I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger, and kept the allowedLateness value to Long.MAX_VALUE. Because of this change the late elements are not discarded and become single element windows Regards, Vinay Patil On Mon, Nov 28, 2016 at 5:54 PM, Sendoh [via

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Yes, That's what I have done. IMO flink should allow user to decide whether to discard late data or not by having some flag, because for some cases users cannot afford to loose any data. Regards, Vinay Patil On Tue, Nov 29, 2016 at 3:16 PM, Maximilian Michels [via Apache Flink User Mailing

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Hi Aljoscha, Yes that it true, but user who are new to flink does not know how to handle the case where they cannot discard late data, so they end up writing custom trigger which was not the case in 1.0.3 Regards, Vinay Patil On Tue, Nov 29, 2016 at 4:29 PM, Aljoscha Krettek [via Apache Flink

Re: Dealing with Multiple sinks in Flink

Hi Robert, I had resolved this issue earlier as I had not set the Kafka source parallelism to number of partitions, so I was getting the issue of window not getting triggered. Now I am facing the same issue, I tried to check the watermark value by using visualVM locally but I am not seeing that v

Re: Dealing with Multiple sinks in Flink

we need to do some configuration locally ? Regards, Vinay Patil On Mon, Dec 5, 2016 at 4:14 PM, rmetzger0 [via Apache Flink User Mailing List archive.] wrote: > Hi Vinay, > > the JMX port depends on the port you've configured for the JMX metrics > reporter. > Did you confi

Re: CodeAnalysisMode in Flink

Hi, Any updates on this thread ? Regards, Vinay Patil On Fri, Nov 18, 2016 at 10:25 PM, Fabian Hueske-2 [via Apache Flink User Mailing List archive.] wrote: > Hi Vinay, > > not sure why it's not working, but maybe TImo (in CC) can help. > > Best, Fabian > > 2016-1

Frequent Full GC's in case of FSStateBackend

after checkpointing , when does this state gets removed / GC. (I am using window operator in which the DTO comes as input) Also why does Flink keep the state in memory after checkpointing ? P.S Using RocksDB is not causing Full GC at all. Regards, Vinay Patil

Re: Frequent Full GC's in case of FSStateBackend

level ? Regards, Vinay Patil On Fri, Feb 10, 2017 at 4:16 PM, Stefan Richter [via Apache Flink User Mailing List archive.] wrote: > Hi, > > FSStateBackend operates completely on-heap and only snapshots for > checkpoints go against the file system. This is why the backend is > typica

Checkpointing with RocksDB as statebackend

Hi, I have kept the checkpointing interval to 6secs and minimum pause between checkpoints to 5secs, while testing the pipeline I have observed that that for some checkpoints it is taking long time , as you can see in the attached snapshot checkpoint id 19 took the maximum time before it gets faile

Re: Checkpointing with RocksDB as statebackend

window operator chain. But as shown in the snapshot ,checkpoint id 19 did not get any acknowledgement. Not sure what is causing the issue Regards, Vinay Patil On Wed, Feb 15, 2017 at 5:51 PM, Ted Yu [via Apache Flink User Mailing List archive.] wrote: > What did the More Details link

Re: Checkpointing with RocksDB as statebackend

Hi Guys, Can anyone please help me with this issue Regards, Vinay Patil On Wed, Feb 15, 2017 at 6:17 PM, Vinay Patil wrote: > Hi Ted, > > I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3 > sink and the 3rd box is window operator followed by chained operator

Re: rocksdb without checkpointing

Hi Abhishek, You can disable checkpointing by not commenting env.enableCheckpointing What do you mean by "We are trying to do application level checkpointing" Regards, Vinay Patil On Thu, Feb 16, 2017 at 12:42 AM, abhishekrs [via Apache Flink User Mailing List archive.] wrote

Re: rocksdb without checkpointing

Good to hear that. On which machine you are running your Flink Job, also what are the configurations you have used for RocksDB I am currently running on C3.4xlarge with predefined option set to FLASH_SSD_OPTIMIZED Regards, Vinay Patil On Thu, Feb 16, 2017 at 10:31 AM, abhishekrs [via Apache

  1   2   >